Six years building a BI practice from scratch

The first assignment sounded small. The consulting firm I had joined, which serves private schools in Brazil, needed a place where clients could see the results of our work. At the same time, I started moving every report the company produced into Microsoft Power BI. Those two jobs, done side by side, became the seed of a BI practice that would keep me busy for the next six years.

This post is the map of that journey, one year at a time. Each year has its own deeper article in this series. Read this one first if you want the whole arc, then jump to the year that looks like the problem on your desk.

Year one: a client area and a bet on Power BI

The client area needed something worth opening, and static numbers were not it. Power BI gave us interactive views that clients could filter and explore on their own, and they noticed the difference fast. By the end of the year we had grown 50%, and BI was no longer a by-product of the consulting work. It had become a product in the portfolio.

That year was mostly learning by shipping. I explain how it went, and how I would plan it today, in Year one: turning Power BI into a product.

Years two and three: leaving Excel behind

In year two the data still lived in Excel files, and the cracks were showing. Spreadsheets have obvious limits on size and access, and every time someone needed to cross two sources, the work became painful for whoever was on the other side of the dashboard. So I moved the bases to SQL Server. Maintenance got easier, handling the data got easier, and security improved a great deal.

A single database also opened questions we could not answer before. One of my favorites: is a school's tuition in line with the families who live around it? We crossed demographic data for the neighborhood with economic data for the city, within radii of 2 and 4 km, to estimate the average amount spent on school fees in the area, and then compared it with the client's own price.

Year three was the rebuild. Every report was redone in Power BI on top of the new SQL Server base, and the semantic models got much faster. We hired an assistant, and while the rebuild went on we kept answering client requests, building specific areas and tailored solutions.

One sentence still sums up those two years: a beautiful dashboard that takes minutes to render does not work. The full story is in From Excel to SQL Server: when spreadsheets hit a wall, and the design side of it in Dashboard UX starts with speed, then storytelling.

Year four: when the volume outgrew the team

Year four was the decisive one. The complexity and the amount of work went past what we could handle. The answer was to move data processing to Python, with pandas and numpy doing the heavy lifting.

Before that, we could process at most three client surveys a day. After the automation, we were handling dozens, and the human work shrank to fixing errors and adjusting questions and answers when something did not fit. That jump came from automation, not from hiring. How we got there is in Python for BI: from three surveys a day to dozens.

Year five: a web portal around the reports

With the processing automated and the team grown to four people, the next step was the experience of the person reading the reports, and taking the product beyond the consulting engagement itself. I built a web application in Python with Flask, a front end in JavaScript, HTML and CSS, and Power BI Embedded to share the reports inside it.

Maintaining reports became simpler, and clients had one secure, practical place to reach their information. Over that period we grew 400%. The architecture and the reasoning behind it are in A client portal with Power BI Embedded.

Year six: from building to teaching

By year six the BI area was standing on its own, and my job shifted. I spent more time leading and training the team than building, setting up continuous training so that everyone kept up with good practice and new technology as demand kept growing. It is the least technical chapter of the six and probably the one that mattered most for keeping the rest alive. More in Leading a BI team through hiring, training and AI.

What the six years taught me

Looking back, the order of the steps says as much as the steps themselves.

1. Visible value first, foundations right after. We moved to Power BI before we had a proper database. Clients saw the difference in year one, and that bought the time to fix the base in years two and three.
2. Speed is part of the content. A correct number that takes minutes to show up loses to a rough number that shows up now. Everything comes down to the experience of the person using the report: the right information, available, with no delays and no bugs.
3. One base changes the questions you can ask. The tuition comparison only became possible once demographic, economic and client data sat in the same place.
4. Automate before you hire. Going from three surveys a day to dozens came from Python, not from headcount. The people we did hire could then work on what needed judgement.
5. The delivery channel is part of the product. The portal made maintenance simpler for us and access simpler for clients. The biggest growth figure of the six years came from that period.
6. A practice lasts as long as its people keep learning. Training is what turned a set of tools into a team that could carry them forward.

Each step made the next one possible. Doing them one at a time, in that order, is what kept the practice moving.

If you are at one of these stages and want to compare notes, or want to see what bi19 builds today, take a look at the bi19 BI platform.