Data and its utilization for a functional, efficient, and sustainable city have always been key concerns of the Smart City community. Beyond mere declarations of intent, however, there was hardly any progress for many years in building cross-departmental data infrastructures within administrations. That is now changing: among other things, supported by the Federal Smart City funding program, an increasing number of municipalities are embarking on establishing professional data management systems. This provides ample reason for us to take a closer look at what comprehensive data management for public administration could look like.
Data Infrastructure – The Foundation of a Smart City
First of all: What exactly is meant by data management, and why is it so challenging? Cities and municipalities accumulate large volumes of data—sometimes as a by-product, sometimes through targeted surveys. This includes, for instance, various types of geographic information, traffic and mobility data, data on energy and water supply, weather and environmental data, data from decentralized sensor networks (e.g., on air and water quality), educational, social, and economic statistics, and not least, data generated by administrative processes (such as during application or specialized procedures). Typically, these datasets are not collected centrally; rather, they are stored in different departments according to responsibility and in a wide variety of formats and structures. Preparing them so that they are findable and usable across departments, while avoiding redundancies and simultaneously protecting sensitive data from unauthorized access, is what falls under the umbrella of municipal data management.
The Basis for Open Data and Efficient Administrative Work
At CityLAB and the Open Data Information Office (ODIS), we have been engaged for many years in preparing and utilizing administrative data as Open Data. With a broad range of formats and resources, we support the Berlin administration—for example, in implementing the Open Data Ordinance and the Open Data Strategy published in 2023. This strategy, in addition to emphasizing the need to provide open data to third parties, also focuses for the first time on improving internal administrative data management. After all, this is the foundation for enabling administrations to share high-quality, up-to-date data with the public and to make administrative work more efficient.
Municipal Data Platforms – One for Everyone?
At first glance, the idea of simply storing all administrative data in one central location appears obvious. However, upon closer inspection, it quickly becomes unrealistic. The still extremely heterogeneous IT landscape of public administration is accompanied by differing processes, requirements, responsibilities, and dependencies. While some administrations have long been operating their own data platforms specifically tailored to their needs (with pioneering examples often coming from departments such as Environment, Traffic, or Urban Development, as well as statistical offices*[1]*), in other places Excel files or PDFs are still being exchanged via email.
A more plausible—and in many cities already emerging—approach is that of central platform systems which connect existing decentralized infrastructures via open interfaces, making them searchable and extending them with additional visualization and analysis tools. This approach has given rise not only to a lively GovTech market but also to development communities in which numerous municipalities pool their resources and drive the development of modular open source suites for data management. Examples include the development community of the Civitas Connect e.V. association and the community around the Urban Data Space Platform, developed together with the FUTR Hub of Tegel Projekt GmbH. The idea is that existing platforms and procedures can be connected to the open system, equipped with metadata, and prepared for further use. In addition to systems for API and rights management, Civitas’ core platform relies on proven tools such as Grafana or Superset to present data in an understandable way for decision-makers and the public.
ETL Processes and Data Pipelines
When data is transferred from several source systems to a target system, it is often sensible to transform it along the way—for example, to improve data quality, standardize the data, or enrich it in other ways. For these so-called ETL processes (Extract, Transform, Load), useful open source tools such as Node-RED exist, which facilitate linking systems and creating data pipelines via a graphical user interface. Nevertheless, effective use of these tools requires at least rudimentary programming or scripting skills, as well as a solid technical understanding of databases and data formats.
Map Applications – Showing the Way!
Georeferenced data offers further visualization possibilities, for instance in interactive maps that can be used in participatory processes in urban development or to create clear and informative presentations. In this context, the master portal developed in Hamburg under an open source license is increasingly becoming the administrative standard, especially as the software is continuously expanded and harmonizes quite well with other open source components.
The versatility of geodata in developing digital offerings can be observed in numerous CityLAB and ODIS projects: initiatives such as “Gieß den Kiez”, the Christmas Market Finder, the Brachenpuls, and many more would not be possible without open geodata.
Metadata Catalogs – Bringing Order to Chaos
To make data repositories findable and reusable, metadata is usually required—information detailing exactly what is contained in a dataset, when it was last updated, who is responsible for it, and under what conditions the data may be used. In Germany, the DCAT-AP model has established itself as the standard for administrative metadata. With metadata catalogs such as CKAN, even larger data repositories can be made easily searchable, which can significantly ease administrative work.
Conclusion: All-in-One Miracle Data Platforms?
In the end, our discussion leads to the insight that no single “perfect” technical solution for cross-administrative data management is in sight. The problem is simply too multifaceted. The most promising approach lies in a well-orchestrated interplay of various components. For the essential elements needed to build a municipal data management system, tried and tested solutions have long been available. The challenge lies in linking these components into a functioning system while simultaneously developing the right structures, processes, and competencies within the administration. This is exactly where we step in and are pleased to accompany the process of testing and establishing an urban data platform.
Administrations must learn to recognize and harness the inherent potential in data. This requires, on one hand, the development of key technical skills and, on the other hand, the establishment of governance models for urban data—which so far exist at best in rudimentary form. In our experience, it makes sense to take these steps directly from practice, following real-world use cases and prototypes. At the same time, these steps demand central strategic control and the sustainable establishment of cross-departmental processes and structures.
[1] In Berlin, in addition to the official state open data portal, there are several other publicly accessible data portals—for example, the Geoportal, the Digital Platform for Urban Traffic, the Social Information System, as well as the pages of the Office for Statistics Berlin-Brandenburg.
