Introducing a framework to help data teams build a documentation-first culture

Photo by Wesley Tingey on Unsplash

“What does this column mean?”
“There are two tables with the same name? Which one can I use?”
“Why are our MRR numbers different in the finance and sales reports?”
“Who can explain our marketing data?”

Messages like these seem familiar? Welcome to the life of every data practitioner in the world.

In 2016, I was at the helm of a data team that was rapidly scaling. We had vast amounts of new tables getting generated, new projects going live, and several new team members joining. But our team wasn’t built to scale.

We thought it would be easy enough to figure this out, but we couldn’t have been more wrong.

Photo by Tom Hermans on Unsplash

At Atlan, we started out as a data team ourselves, driving data projects for social good with organizations like the United Nations, Gates Foundation, the World Bank, etc.

We acted as a “data team” for our customers, so we experienced all the chaos and frustration of dealing with large-scale data firsthand. We were awoken with crisis calls every couple of days for even tiny issues — whether it was troubleshooting why a number on a dashboards was incorrect to trying to get access to the right data set.

We worked with a wide variety of data, everything from 600+ government…

A beginner’s guide to the best of breed tools and capabilities for your Data Platform initiative

Photo by Adam Nowakowski on Unsplash

If you Google “modern data platform”, you’ll immediately be bombarded with advertisements and lots of companies professing that they are the one true data platform. Not so helpful, right?

So what the heck is a modern data platform? What does that even mean, and what does it look like in 2021?

The short answer: a modern data platform is a collection of tools and capabilities that, when brought together, allow organizations to achieve the gold standard — a fundamentally data-driven organization.

In this article, I’ll break down what a modern data platform means in practice today. This includes the three…

Modern metadata solutions, data quality frameworks, infrastructure, job roles, and other big changes are on their way.

Photo by Chris Liverani on Unsplash

Just like every other realm, 2020 upended the data world. When COVID shut down businesses and sent employees to work from home, companies had to quickly adapt to the “new normal”.

Cloud became an absolute necessity, as organizations moved to working remotely. Data governance and security became a big priority, with everyone accessing data from different locations and systems. Smarter AI became appealing, now that historical models were meaningless. In short, organizations realized they needed to make changes fast. Data investments went up and organizations sought to upgrade their systems and create the perfect data stack.

With 2020 in the…

It’s time for a modern metadata solution, one that is just as fast, flexible, and scalable as the rest of the modern data stack

Photo by Haneen Krimly

2020 brought a lot of new words into our everyday vocabulary — think coronavirus, defund, and malarkey. But in the data world, another phrase has been making the rounds… the modern data stack.

The data world has recently converged around the best set of tools for dealing with massive amounts of data, aka the “modern data stack”. This includes setting up data infrastructure on best-of-breed tools like Snowflake for data warehousing, Databricks for data lakes, and Fivetran for data ingestion.

The good? The modern data stack is super fast, easy to scale up in seconds, and requires little overhead. The…

As SocialCops reaches the four-year mark, we have been thinking deeply about the “dent” our company could leave on the universe. We’re asking ourselves, “What would it take to create a company that would have relevance for hundreds of years?” At four, SocialCops has lasted 36.5 months longer that the average startup in India. We are moving rapidly from our childhood days into adolescence — an interesting time, where one typically has a strong sense of self and is now battling to grow into the adult one is meant to be.

In the quest to shape our future self, we…

When we started SocialCops, my co-founder & I were 21 and fresh out of university. This was before startups were sexy in Asia! We had no real startup background, had interned in investment banks and large tech giants — had never worked in a startup or seen one grow. Yeah, we read TechCrunch all the time and watched Startup School videos but as Drew Houston once said, the only thing that can ever really prepare you for a startup is possibly working at a startup. So, in a nutshell — we had a lot to learn.

An avid reader/geek, I…

In the past few years, I’ve spent a lot of my time interacting with the “next billion” and using technology and data to solve problems that are at the core of development — in urban and rural india.

Let me back up a little bit —I’m Prukalpa, the cofounder of SocialCops. At SocialCops, we’re building a data platform to tackle some the biggest problems facing humanity. In the short 2 years of our existence, our data platform has been used to drive healthcare policy, optimise disaster relief operations and solve urban planning issues. …

It’s that time of the year when the entire world is congratulating you about all the jobs that you’ve landed. You’ve done a great job at college — as president of your students union, spending your summer and winter breaks at super cool internships and commercializing the product that you built as part of your research project. You spent your university days organizing the largest scale of tech fests, leading large teams, wearing multiple hats and building cool projects. The world says it’s time to sit back and relax. It’s your final semester.

But you’re feeling uneasy. Somehow the jobs…

Prime Minister Modi at a Nasscom event on 1 March. Photo: Subhav Shukla/PTI

Globally, governments have often collaborated with start-ups to fast-track innovation.

For instance, it would be impossible to visit the Kennedy Space Center in Florida, US, without hearing government officials speak admiringly of Space Exploration Technologies Corp. (SpaceX), an American space transport services company — the only private company to bring back its spacecraft from low-Earth orbit.

George Tenet, a former head of the Central Intelligence Agency (CIA), has been quoted by Forbes magazine as saying that he wishes data technology start-up Palantir Technologies had been in existence…


Cofounder @atlanHQ — Modern Collaborative Workspace for Data Teams @forbes 30u30 and @fortune 40u40, @TED speaker

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store