HOTOSM recognition by the President of Mexico in Internet Day 2016

On the Internet Day, May 17 2016 the President of Mexico Enrique Peña Nieto invited fifty citizens who called Digital Leaders (#LideresDigitales) to have a dialogue on the future of technology and Internet in Mexico, I had the opportunity of being among these group of citizens. The President talked about various related topics but especially appreciated the efforts of humanitarian mapping conducted by Humanitarian OpenStreetMap in Hurricane Patricia.

Day of Internet 2016- Dialogue with the President of Mexico
día del internet_3
President’s office twitter account

Why the President of México thanked the efforts from HOTSOM in Hurricane Patricia? Here you will find some details so you know what was done.

On October 23, 2015 Hurricane Patricia threatened to touch the states of Colima, Nayarit and Jalisco with winds up to 325 km/h, authorities of Mexico mentioned “It is very likely that this hurricane is the strongest ever in the Pacific side of our country, since it has records ”

Hurricane Patricia Path from NOAA

Contributors of OpenStreetMap and HOTOSM like Rodolfo Wilhelmy, Humberto Yances, Rafael Avila, Robert Banick, Andres Ortiz, and many others (sorry for not mentioning everyone)  in addition to an army of over 500 mappers of Mexico and the world joined efforts to support this area of the Mexican Pacific with data that could be used for the benefit of the population that could be affected. Fortunately, the hurricane lost strength by touching the coast of Mexico causing minimal damage compared to what was expected.

Quick stats:

  • More than 500 contributors mapped 9,000 kms of roads (5.6k miles of road) + 72,000 buildings in 72hrs
  • It was processed 29,608 km^2 pre-event DigitalGlobe imagery to improved coverage over priority areas.
  • It was analyzed INEGI road data to identify missing roads and road names in OSM data.
  • Mexico Open Data was confirmed by authorities to be used in OpenStreetMap.

All these was possible thanks to the great work HOT members, companies supporting OSM project and the local community in Mexico and the World

Contributors mapping the priority area in 72 hours Gif by Mapbox

The event took place in Los Pinos (The equivalence of the U.S. White House) at the moment the Open Data topic was mentioned, Peña Nieto said he knew someone who had supported the alert for Hurricane Patricia was among the guests so I raised my hand to start the dialogue, the President mentioned “… I just want to thank because it’s an example that illustrates very well what we can achieve and I think that you also use open information.” In my participation  I could give my point of view on the need for Mexico not only upload open data to be the first in quantity of released Open Data but emphasize the need of quality Open Data in order to take better decisions based on them. Also I could mention the importance of Open Mapping and collaboration between Governments and Civil Society so more Mexicans are less harmed by disasters (find the video here).

Miriam_DiadelInternet
Fragment of video from President’s office twitter account

In Mexico the OpenStreetMap community is not as numerous as in other countries but in the last two years a group of collaborators we have joined together to promote the project and increase the local community through massive workshops in Universities and courses for Government Authorities and Civil Society. Much remains to be mapped but I believe we are on the right track.

Miriam Gonzalez

@mapanauta

Facebooktwittergoogle_plus

Happy Mapping Hour – Presentation Import Project INEGI MGN (National Geostatistical Framework)

Last April 6th 100% of the Mexico Telenav’s team (Andrés Ortiz 50% and Miriam Gonzalez 50% 😀 ) presented the results of Import Project INEGI National Geostatistical Framework. The meeting point was the Felina bar on the edge of Condesa and Escandon neighborhood.

Andres_presentando
Andres presenting at Happy Mapping Hour
Miriam_presenta
Miriam presenting in Happy Mapping Hour                                                         Image by @Tlacoyodefrijol

More than 20 people booked and came to the appointment. The project was originally announced in May 2015 with much skepticism because this was the first time a project of such magnitude was taking place in Mexico and the OpenStreetMap community in Mexico at that time was very disperse.

Before_import
Before the Import Project there were 69 valid boundaries                               Image by Ruben @Mapbox

Many import projects have been conducted in many parts of the world, such projects have helped (mostly) to create the map of the world that we have today and Mexico was going to be part of them. People with extensive knowledge in imports formed part of the project including Victor Ramirez, Ernesto Carreras, contributors od OpenStreetMap Puerto Rico and Rafael Avila, a HOTOSM collaborator and expert in African countries imports. At the beginning of the project we realized that there were only 69 valid administrative boundaries (although in the image it looks more than 69,  these lacked the tag SOURCE which made them invalids) and the end of the Import project the team had added 2,457 administrative boundaries with tag Source = INEGI MGN 2014 v6.2

After_Import
After the Import Project there were 2,457 administrative boundaries        Image by Ruben @Mapbox

To the #HappyMappingHour diverse OSM contributors atended such as geographers, developers, archeologists and also Armando Aguiar – INEGI IT Services Director  witnessed how the Open Data Inegi released at the end of 2014 has been in benefit of OpenStreetMap. Let me share some statistics:

Quick Statistics:

Node numbers/Ways/ Deleted relations

  • 500K / 2k / 500

Node numbers/ Ways / Added relations

  • 1000K / ~4k / ~1050

Number of hours dedicated :

  • 250+

NUmber of administrative boundaries added:

  • 2,457

 

Now that the map has de MGN boundaries as a reference mappers as Irk_Ley have been investigating the local laws of the states of Veracruz and have been reviewing historical maps of the Map Library Manuel Orozco. These mappers will be verifying and correcting those limits which have differences with the MGN when they have the backup of the documentation of the local law.

Ancient map of the Papatla, Veracruz region

Here you will find the presentation of #HappyMappingHour and if you want more technical details we suggest you check the following blogs and the wiki.

Here also you will find two Blogs from collaborators in the Import Project:

Blog: My experience in OSM during the MGN Import by Pablo Garcia (OSM user: Irk Ley)

Blog: Import of INEGI Mexico municipalities finished by Andres Ortiz (OSM user: Andresuco)

You can contact them directly if you have any questions or comments for them.

What are the next challenges?

Evaluate data from the National Road Network and create a joint project with Mexico OpenStreetMap community to carry out its import. It is also in the radar create a tool where information from OpenStreetMap in Mexico is a kind of “inspector” to send feedback to INEGI about possible shortcomings or errors can be corrected and improved thanks to contributors OpenStreetMap but first we need more discussions with the local community.

Facebooktwittergoogle_plus

Using PostGIS to answer geodata questions

One of the biggest challenges when working with large sets of data is to find the least costly workflow that you have to follow in order to get the most accurate answers.

Let’s say you have a huge dataset composed of all sorts of geometry features (points, lines, areas etc.) and you want to do a bit of cleaning – because messy and redundant information is no fun!

So you might be thinking “Hmmm… which are the areas that have an unnecessary high density of points?”

The same issue can arise when working with OpenStreetMap data. This can be easily solved using PostGIS and a command line tool that we’ve created and using.

Note: The following steps require a Linux environment, Postgresql 9.x, PostGIS 2.x, Osmosis 0.43+, QGIS 2.12.2+

Getting the data

Download an *.osm.pbf file using command line:

 wget https://s3.amazonaws.com/metro-extracts.mapzen.com/san-francisco_california.osm.pbf

This is the metro extract for San Francisco, provided by Mapzen. Geofabrik is also a very good resource for OSM data extracts.

In the same folder, download SCOPE – databaSe Creator Osmosis Postgis loadEr.

wget https://github.com/baditaflorin/osm-postgis-scripts/blob/master/scope.sh

Make sure to set the file to be executable by using

chmod +x scope.sh

Load the data

Using SCOPE and following the instructions on the screen, load the *.osm.pbf into a database.

SCOPE automatically creates the database with hstore and postgis extensions and the pgsnapshot schema.

Play with the data

Now that you have the data set up, you can easily query it using the DB Manager from QGIS and some PostGIS scripts.

Interesting examples

For example, using the find_duplicate_nodes query, we can see that this building (@20.805088941495338, -104.92877615339032), appears on the same spot 23 times!

duplicate_building

The one next to it (@20.8054225, -104.9278152) appears 22 times!

duplicate_building_2

The node density for these areas (@20.4411867, -97.3172739) is too high – 168 nodes!

nodes1

Also, 171 nodes for a small fence segment (@46.7487683, 23.559687)!

fence

the-node-density

Feel free to fork the GitHub repository and modify the code to suit your needs! Also, if you feel insipred, you can suggest a better and shorter name or acronym for SCOPE!

Facebooktwittergoogle_plus

OSM Mapping party – spring edition

 

On the 17th of April we had our first Mapping party event for this year. Our main focus was to improve the map of our hometown by reflecting the latest changes.                                                                                                   Cluj-Napoca is a dynamic city, many new buildings was constructed; POIs, turn restrictions, addresses have been changed and appeared since the last field mapping.

Around 30 map enthusiast show up in Sunday morning for the Mapping party. There were both experienced mappers and newbies present at the event. The event had started with a morning coffee and some instructions regarding data collections.

For data collections we used the following tools:
• Field papers: our colleague Florin Badita had took some time before the event and had created field papers for several city areas

field-papper1

• GPS tracker applications: OSMTracker, OSMAnd, Pushpin OSM and so an
OpenStreetView application 

We have divided the people into smaller groups of 2-3 persons. After each group had chosen an area to map we went out to collect the data.
On the afternoon we headed back to our meeting location to add the collected data into OpenStreetMap.

An outcome overview of our mapping effort is presented on the following images:

FinalEdits

FinalEditsOverview

Facebooktwittergoogle_plus

Turn restrictions – a vital part of any routing system

The best part of using everyday OSM technologies and relying on OSM to make sure that you get “there” on time is that you can directly influence the quality of the experience.

Regardless which OSM technology you’ll be using, to provide you the best experience possible, the routing software has to know as much information as possible about the roads between you and your destination: one-way streets, turn restrictions, speed limits, road closures and much more.

For example, the turn restrictions contribute significantly to the total travel time, and to the correctness of the route altogether, thus, by ignoring them in the traffic network model, essential characteristics of the network might be missed, leading to substandard and unreasonable paths.

Dealing with turn restrictions in OSM

To help us navigate the complexities of properly translating real map scenarios to the ways and points schema of OSM we will rely on JOSM with the turn restrictions plugin installed.

Turn restrictions in OSM are handled by creating a relation

A relation is one of the core data elements that consists of one or more tags and also and ordered list of one or more nodes, ways and/or relations as members which is used to define logical or geographic relationships between other elements. (source)

There is a mandatory requirement when creating a turn restriction relation: it has to consist of minimum three members and must have assigned two tags. (see below example)

structure
The ‘type=restriction’ flags the relation as a turn restriction and ‘restriction=no_u_turn’ indicates the restriction type.

A ‘no_’ type relation can also be represented in map data as an ‘only_’ type relation. The prohibited turn restriction relation is preferred by some routing engines instead of an allowed turn restriction relation.

More details here - https://wiki.openstreetmap.org/wiki/Relation:restriction
More details here – https://wiki.openstreetmap.org/wiki/Relation:restriction; US regulatory signs – http://mutcd.fhwa.dot.gov/services/publications/fhwaop02084/

Members of a turn restriction relation are ways and nodes

One simple case can be a turn restriction relation that consists of three members – two ways and one node. The two ways would represent the beginning (‘from’ role) and end (‘to’ role) of the turn restriction. The node would represent the continuity of travel between two ways and has a ‘via’role.

Way (A) - node (B) - way (C) sequence
Way (A) – node (B) – way (C) sequence in a ‘no_left_turn’ restriction relation.

Another case is where a turn restriction relation can consist of three or more ways. Two ways from this type of relation would represent the beginning and end of the turn restriction and at least one way would represent the continuity of travel between the aforementioned ways (‘via’ role).

Way (A) - way (B) - way (C) sequence in a no_u_turn restriction relation
Way (A) – way (B) – way (C) sequence in a ‘no_u_turn’ restriction relation.

Workflow for adding turn restrictions

The traditional way

Using the embedded relation editor available in JOSM. A slight disadvantage of this method is that you spend a bit more time to manually construct the relation. Click on the image below for how-to video.

traditional_way_vid

The user-friendly way

Using the turn restrictions plugin, that automatically recognizes the type of relation and roles for each member. Click on the image below for how-to video.

user_friendly_vid

Using the aforementioned tools, we have reviewed 2,000 miles of field trip footage and added nearly 2,500 turn restrictions in the LA/Orange county area, where 85% of the turn restrictions that were added to the map are no_u_turns, followed by 11% of no_left_turns, the rest being covered by the other categories.

Hopefully we’ve managed to illustrate how easy is to map turn restrictions in OSM. Now, it’s your turn!

Facebooktwittergoogle_plus

How we imported Administrative Boundaries for Mexico from INEGI

The INEGI boundaries import project is focused on importing the data of the national, state, municipal and sub-municipal level divisions present in the MGN published by the INEGI in a community monitored process.

One of the current problems in OSM regarding Mexico’s data is the incompleteness of the administrative boundaries for municipalities. Municipalities are the second-level administrative division in Mexico, the first being the state. There are 2456 municipalities, including the ones in Mexico City which are also a second-level division just with a different name – delegations.

The main goal of this process is to enhance the current OSM administrative division coverage of Mexico with open data made available by the government at the end of 2014.

Import Process

The following steps describe the entire workflow we followed to import the boundary data.

  • Step 0 – Reprojection of INEGI dataset

Before any other step, the data released by INEGI has to be reprojected to WGS84 (EPSG:4326), from ITRF92, using QGIS, and saved as .shp file. An important thing to mention is that no simplification of the boundary geometries is considered whatsoever for this or any of the subsequent steps since the geometries are official government data.

  • Step 1 – Conversion to OSM data

Download the state boundary of interest relation from OSM and save it as an .osm file. In QGIS, using Vector > Research Tools > Select by Location, select the INEGI municipalities boundaries that are within the area of interest, in this case Quintana Roo state, and export the selection as .shp file.

Municipalities in Quintana Roo state.
Municipalities in Quintana Roo, as polygons.

The exported features will be polygons. In order to process them, they must be converted to lines in QGIS using the Polygons to Lines option, available in Vector > Geometry tools. Visually, the output will look the same as when the municipalities were polygons.

Municipalities in Quintana Roo, as lines.
Municipalities in Quintana Roo, as lines.

Using ogr2osm the .shp file containing the boundaries as lines is converted into an .osm file.

Before moving forward, the resulting .osm file has to be modified a bit. Using Notepad++, open the file and search and replace <nd ref=’ with <nd ref=’- and <node id=’  with <node id=’-, so the file will be with negative id.

The negative id is important because JOSM will know that this is new data, not yet added to the map.

Next, the .osm file can be converted to an .osm.pbf file using osmosis.

  • Step 2 – Processing

We load the .osm.pbf file from the previous step into an internal tool, called Mexico Split. The tool is designed to eliminate duplicate/overlapping ways by detaching them from their parent polygons and replacing them with a single common way of the two involved polygons.

mxsplit3
Detects overlapping ways and replaces them with a single common way.

Besides this main purpose, the tool also splits any resulting ways longer than 2000 segments in shorter ways, groups the ways in relationships according to the borders they define and adds some predefined tags to these ways and relations.

Tags added to relations:

type=boundary

INEGI:MUNID=<value_from_the_original_polygon>

name=<value_from_the_original_polygon>

Tags added to both ways and relations:

boundary=administrative

admin_level=6

source=INEGI, MGN 2014 v6.2

For example, data for Bacalar municipality contains the following information:

Bacalar municipality in Quintana Roo state.
Bacalar municipality in Quintana Roo state. (click for larger image)
  • Step 3 – Backup and metrics of existing OSM data

We took a backup of the current OSM data previous to the import of the regions that are going to be impacted, using Overpass API. Also, tag related metrics have been recorded – source, population, admin_center, admin_label, wikipedia etc. in order to have an overview of the newly added information.

  • Step 4 – Delete existing data from OSM and upload fresh data

In some cases the states already have some information regarding municipality boundaries (admin_level=6). These will be deleted, but before deletion we take a look at all the features and relations, to have a very good image of what we should put back in map data after the import.

Next, we upload the municipalities on a state by state basis.

  • Step 5 – Clean/verify the newly added data

This is a very important step because we verify the data that we’ve uploaded to make sure that there are no errors and manually re-link the admin_level=6 relations to the admin_level=4 boundaries, where required. Any other manual corrections are done at this step.

output_mSaeg1
An example of the newly added municipalities boundaries in Tabasco.

To ease the process of importing the municipality boundaries, we use the Mexico Import Map paint style for JOSM. It highlights the last node of every way, making it simple to see the length of every way.

Map styles - JOSM default vs. Mexico Import
Map styles – JOSM default vs. Mexico Import. (click for larger image)

The square node also has a certain degree of transparency, so we can see if there is a node under the node. To be able to work in a systematic way, it allows to quickly see duplicated nodes and see the difference between the admin_level=4 and admin_level=6.

Facebooktwittergoogle_plus