Is OpenStreetMap Big Data ready?

This article was written by Adrian Bona as a draft for a talk at State of the Map US in Boulder, Colorado this past month. The talk did not make it into the program, but the technology lives on as a central part of our OpenStreetMap technology stack here at Telenav. We will continue to deliver weekly Parquet files of OSM data. Adrian has recently moved on from Telenav, but our OSM team is looking forward to hearing from you about this topic! — Martijn

Getting started with OpenStreetMap at large scale (the entire planet) can be painful. A few years ago we were a bit intrigued to see people waiting hours or even days to get a piece of OSM imported in PostgreSQL on huge machines. But we said OK … this is not Big Data.Meanwhile, we started to work on various geo-spatial analyses involving technologies from a Big Data stack, where OSM was used and we were again intrigued as the regular way to handle the OSM data was to run osmosis over the huge PBF planet file and dump some CSV files for various scenarios. Even if this works, it’s sub-optimal, and so we wrote an OSM converter to a big data friendly columnar format called Parquet.The converter is available at github.com/adrianulbona/osm-parquetizer.Hopefully, this will make the valuable work of so many OSM contributors easily available for the Big Data world.

How fast?

Less than a minute for romania-latest.osm.pbf and ~3 hours (on a decent laptop with SSD) for the planet-latest.osm.pbf.

Getting started with Apache Spark and OpenStreetMap

The converter mentioned above takes one file and not only converts the data but also splits it in three files, one for each OSM entity type – each file basically represents a collection of structured data (a table). The schemas of the tables are the following:

node
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- latitude: double
 |-- longitude: double

way
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- nodes: array
 |    |-- element: struct
 |    |    |-- index: integer
 |    |    |-- nodeId: long

relation
 |-- id: long
 |-- version: integer
 |-- timestamp: long
 |-- changeset: long
 |-- uid: integer
 |-- user_sid: string
 |-- tags: array
 |    |-- element: struct
 |    |    |-- key: string
 |    |    |-- value: string
 |-- members: array
 |    |-- element: struct
 |    |    |-- id: long
 |    |    |-- role: string
 |    |    |-- type: string

Now, loading the data in Apache Spark becomes extremely convenient:

val nodeDF = sqlContext.read.parquet("romania-latest.osm.pbf.node.parquet")
nodeDF.createOrReplaceTempView("nodes")

val wayDF = sqlContext.read.parquet("romania-latest.osm.pbf.way.parquet")
wayDF.createOrReplaceTempView("ways")

val relationDF = sqlContext.read.parquet("romania-latest.osm.pbf.relation.parquet")
relationDF.createOrReplaceTempView("relations")


From this point on, the Spark world opens and we could either play around with DataFrames or use the beloved SQL that we all know. Lets consider the following task:

For the most active OSM contributors, highlight the distribution of their work over time.

The DataFrames API solution looks like:

val nodeDF = nodeDF
    .withColumn("created_at", ($"timestamp" / 1000).cast(TimestampType))
    .createOrReplaceTempView("nodes")

val top10Users = nodeDF.groupBy("user_sid")
    .agg(count($"id").as("node_count"))
    .orderBy($"node_count".desc)
    .limit(10)
    .collect
    .map({ case Row(user_sid: String, _) => user_sid })
    
nodeDF.filter($"user_sid".in(top10Users: _*))
    .groupBy($"user_sid", year($"created_at").as("year"))
    .agg(count("id").as("node_count"))
    .orderBy($"year")
    .registerTempTable("top10UsersOverTime")


The Spark SQL solution looks like:

select 
    user_sid, 
    year(created_at)) as year,
    count(*) as node_count
from 
    nodes
where 
    user_sid in (
        select user_sid from (
            select 
                user_sid, 
                count(*) as c 
            from 
                nodes 
            group by 
                user_sid 
            order by 
                c desc 
            limit 10
        )
    )
group by 
    user_sid, 
    year(created_at)
order by 
    year


Both solutions are equivalent, and give the following results:

alt tag

Even if we touched only a tiny piece of OSM, there is nothing to stop us from analyzing and getting valuable insights from it, in scalable way.

If you are curious about more advanced interaction between OpenStreetMap and Apache Spark, take a look at this databricks notebook.

OpenStreetMap Parquet files for the entire planet?

Telenav is happy to announce weekly releases of OpenStreetMap Parquet files for the entire planet at osm-data.skobbler.net.

Facebooktwittergoogle_plus

New ImproveOSM tiles are ready to be used!

New ImproveOSM missing road tiles are available! The new data is very helpful as they can help you to target the missing roads, add them to OSM and thus greatly improving the map.

Worldwide, there are 113048 new road tiles.  The countries with the highest number of tiles are: Russia – 38669 tiles, United Kingdom – 8890 tiles, Kazakhstan – 10993 tiles, India –  9418 tiles and the United States- 7560 tiles (see graph below). There are few new tiles in Detroit too so that you are welcome to give us a hand with them! You can find more information about our work in Detroit on our blog (http://blog.improve-osm.org/en/2017/08/lane-number-and-turn-lane-editing-in-detroit/).

 

Facebooktwittergoogle_plus

Lane number and turn lane editing in Detroit

Since we started editing in Detroit, we focused on making OSM navigation ready. We started with the basics: road geometry, road name, turn restrictions, and then we were able to further build on this foundation by adding details like lanes and turn lanes. In the last four months, we focused on adding and updating the lane info (lane number and  turn lane) on motorway, motorway_link, trunk, trunk_link, primary, primary_link, secondary, secondary_link roads in Detroit, Michigan.

For editing lanes and turn lanes we used JOSM, the TurnLanes-tagging Editor plugin and the Lane and road attributes map paint style.

We had two kinds of lane editing: unidirectional road editing, bidirectional road editing. The only difference between those two is the direction tag used in the second case, as you can see in the below table:

For every edited case, we used a simple workflow:

  • we split the way where the number of lanes changes
  • we checked and double checked the aerial imagery to make sure we enter the correct number of lanes and add the appropriate lanes tag
  • we opened the turnlanes-tagging plugin and activated the Lane and road attributes map style
  • using the plugin, we selected the type of the road: Unidirectional road or Bidirectional road
  • we marked the number of lanes for each way needed
  • we marked  the direction on each lane
  • before uploading the data, we checked again that the turn lanes that we had added  were similar to the markings on the road!

The approach of the main cases we’ve met during our edits are exemplified in the next GIFs.

Editing the number of lanes

Adding both ways lane

In some particular cases, when there were doubts, we consulted the OSM community on Github and Talk-US.

While editing, we paid special attention to other already existing features (like route relations, turn restrictions, speed limits, etc). Because all Telenav Mapping team was involved in this project, we established from the beginning some rules, in order to have consistency in our edits:

  • Add a new lane only when you have a line marked on the road (use the satellite imagery, OSC photos  to validate the marks).
  • Links without any marks on road or without one way tag should be edited as a bidirectional road, adding one lane on both driving directions.
  • Never add the turn lane before or after the continuous line mark on the road. The turn lane will be added  starting from the beginning of the continuous line mark on the road.
  • We split and edit lane number even when we have small segments of ways.
  • The location of the junction nodes should be at the beginning of the continuous line marks.
  • We always add the yellow both way lane.
  • We DO NOT add the yellow striped lanes and double marked line lanes.

The main sources used during the project were aerial imagery (Bing, Mapbox, NAIP, Digital Globe) and street level imagery: OSC, Mapillary.

We worked on this issue for 2 months and succeeded to review a large part of motorway, trunk, primary and secondary roads from Detroit area, in order to add or update lane info. During this project, we managed to review 3100 miles and edit 1730 miles of roads.

Here’s how the number of miles of roads with lane information has increased during the project:

The edits we made cover a large area of the Wayne, Macomb and Oakland counties. In the GIF below you can see an evolution (difference between March and July) of our lane info edits in OpenStreetMap.

Heatmaps with our edits during the last four months:

When we finished editing lanes and turn lanes in Detroit, we started assessing the general quality of the lane info by using different approaches. Internally, we call this process quality assurance and we think it is vital to do it after the end of each project.

During the QA process we edited lane info on about 400 miles of roads, and the main issues that we corrected were:

  • incorrect number of lanes and turn lanes
  • duplicated/overlapping ways
  • missing both way lane
  • oneways with lanes:forward/lanes:backward info
  • check roundabouts to have the proper number of lanes

Below you can see some examples of our improvements:

 

Facebooktwittergoogle_plus

Find your MapRoulette Challenge

MapRoulette is a fun way to spend a few minutes (or hours…) improving OpenStreetMap. MapRoulette will present you with a random, easy to solve issue in OSM. MapRoulette is organized in ‘Challenges’, groups of tasks that are of the same nature. For example, there is a challenge to add missing crosswalks in various areas in Switzerland, based on analysis of aerial images.

How do you find a challenge you would like to work on? The MapRoulette home page provides a map of all the challenges, but this has some shortcomings. The challenge ‘centers’ are no

t always representative of where the tasks actually are located. It is also hard to search by topic. MapRoulette also has a search bar that you can use to find a challenge by keyword.

I want to work on making it much easier to find

 interesting MapRoulette challenges, and I would like to hear from you how you think that should work. Please add a comment below with your ideas!

In the mean time, I made a page that lists the most popular and newest challenges. It is a bit of a hack so let me know if it stops working 😉

Happy Mapping!

Facebooktwittergoogle_plus

Improving OSM in Canada one day at a time

Ever since we started our mapping project in Canada, nearly 8 months ago, we’ve been continuously working on bringing the OSM data to the level where all elements needed for routing get as detailed as possible.

Whether we are talking about the basics of road networks such as geometry, naming or traffic flow direction, to in-depth details like number of lanes, turn lanes, turn restrictions, signposts and even complex relations referring to highways, we edit everything.

Our main focus is oriented towards the Top 5 metro areas: Toronto, Montreal, Ottawa, Vancouver, Calgary. These are the places where we spent the most of our time researching for open data, adding new features, editing existing ones. In order to make sure that the overall state of OSM throughout the entire region of Canada is in navigable ready state, we’ve also included the first 50 cities based on population.

So, let’s see some numbers and graphs because everybody likes those. If we start looking at the numbers for the entire region we can see a significant rise in road geometry that was added, around 3% (25,330 miles) out of the total numbers of miles. The same goes for roads that previously did not have name tags with a rise of little over 3.5% (16,799 miles).

A more significant change can be noticed for features that weren’t extensively mapped before in the area, such as turn restrictions rising from 5254 to 54891, or signposts which hadn’t been mapped under the same standardized method. With the help of OpenStreetCam and Mapillary pictures, we’ve managed to add relevant signpost information increasing the number of nodes well over 68%.

If we break down the numbers for the Top 5 areas, the most noticeable changes can be observed for both Toronto and Montreal where oneway tags and signpost information have been improved.

One of our main goals is to focus not only on quantity but especially on quality. This is why we have multiple tools for integrity checking that are ran periodically on the entire region of Canada. These tools cover a wide variety of cases that are being corrected weekly, such as: road name flip-flops, unconnected ways, smoothness problems, misnamed road, road names having their suffixes or prefixes abbreviated and many more.

We make use of different QA tools (KeepRight/Osmose) to search and track issues in OSM that have either been added by mistake or have remained unedited after large imports. We’re also on the look out to improve way accuracy and fix alignment issues.

An overview of our edits.

Below you can see some examples of our improvements.

Road geometry updates.
Road geometry alignment.
Missing geometry and minor refinements.
Turning loops updates.
Facebooktwittergoogle_plus

OpenStreetCam JOSM plugin – new features

Last week we had released a new version of the OpenStreetCam JOSM plugin. While we are continuously working on improving and fixing existing functionality, we also keep adding new and exciting features.

Map view improvements

This new release introduces a major improvement to the map view. For small zoom levels, we had adopted a similar visualization as in the case of the web and mobile OpenStreetCam applications. Instead of displaying individual photo locations we display ways that have OpenStreetCam data coverage.Segments are colored with purple and have different transparency based on the data coverage: segments that have many images are opaque while the segments that have only a few images are more transparent.

By changing the initial MapView visualization we were able to display OpenStreetCam data starting with zoom level 10. This way we can indicate areas that have street view coverage at a country view level and possible gave a hint to the user where he/she can find an extra source of mapping support.

Starting with zoom level 18 the map view changes and individual photo locations are displayed similarly as in the previous versions of the plugin.

The displayed data type is user configurable and can be changed from the OpenStreetCam plugin preference settings. You can access the preference settings from JOSM ->Preferences -> OpenStreetCam plugin -> MapView settings or from the OpenStreetCam panel by clicking on the preference icon. 

From the MapView settings section, you can change the minimum zoom level at which image locations are displayed, along with the data type change method. By default, the MapView data type is changed automatically.                                                                                               When the “switch manually between segment and image view” option is enabled a new button is visible in the OpenStreetCam panel.

The “data switch” button is enabled starting from zoom 16 and is represented with different icons based on the displayed data type. For segment map view a photo icon is displayed while for image location view a segment icon.

If you click on the button the map view changes from segment view to image location view and vice-versa.  The type of data can be changed manually for zoom levels bigger than 16.

 

Layer and panel improvements

The OpenStreetCam layer and panel default visibility had been improved and previous open/closed states are remembered for future JOSM sessions. After installing the plugin in order to see the OpenStreetCam data you need to open manually the layer and panel. The layer can be opened from Imagery ->OpenStreetCam menu, while the panel from the left side JOSM menu.

We had changed the OpenStreetCam window button panel actions and removed the actions that were not related to the currently selected image. Feedback and filter actions were added to the OpenStreetCam layer menu:

In case you need a refresher:  OpenStreetCam data can be filtered based on date and currently logged in OSM user. Basically you can visualize images that were uploaded after the specified date. You can also visualize only your contributed data.

Nearby photos

An important feature that we had added to the plugin is the nearby photos functionality. This functionality improves the mapping process especially if the selected photo does not contain all the information or if the selected photo has bad quality or has not the right angle.

A nearby photo of a selected photo can be visualized either by clicking on the “Nearby photo” icon or by pressing ALT+N keys. 

If the “Load track on image selection” preference settings option is selected , than also the track corresponding to the nearby photo is loaded.

Nearby photos are computed based on the currently visible photos, if the user moves the map or zooms in the set of nearby photos is recomputed.

A photo is considered nearby if belongs to a different track and it is located to maximum distance from the selected photo.

Photo load on mouse hover

Another important feature that we had added to the latest release allows users to quickly load photos on mouse hover action.  By default this feature is disabled and can be activated from JOSM ->Preferences -> OpenStreetCam plugin -> Image settings.

If this feature is activated, than the small thumbnail image is loaded in the OpenStreetCam panel and remains loaded only it is explicitly un-selected from the map.

A better resolution image is loaded if you click on the image location icon or if the OpenStreetCam panel is maximized.

Upcoming features

The JOSM plugin is work on progress, we are working on improving the usability and plan to add new features from time to time.

We hope that you enjoy the new features! If you have ideas, suggestions or encounter any issue with the plugin during editing sessions please submit either to the GitHub issue page or to the Feedback forum .

Have fun improving the map by using OpenStreetCam images!

Facebooktwittergoogle_plus

OpenStreetCam JOSM plugin

The OpenStreetCam JOSM plugin helps the community to improve the map by displaying up to date street view images. Street view images are collected by the OpenStreetCam platform and are available also via the OpenStreetCam web application and map editor.  

Having an extra source of free and open imagery ease the process of remote mapping and allows the users to reflect the reality also in the map. Street view images are helpful for editing map features that are not visible on satellite imagery like traffic signs, house numbers, bus stops, points of interests.

Installation

Install the OpenStreetCam plugin the familiar way, through the JOSM plugin Preferences menu item. After you install the plugin and restart JOSM, you should see the OpenStreetCam layer and panel.

OpenStreetCam layer

After a successful installation the OpenStreetCam layer is available in the layer menu panel and on the main map the image locations are displayed. Image locations are illustrated with blue icons, each icon indicating the image heading.

An image location can be selected by single mouse click action as long as the layer is visible. You can select images even if the OpenStreetCam layer is not the active layer.

OpenStreetCam layer displays data starting with zoom level 14, so in order to see the data you need to zoom in into the desired mapping area.

For Imagery layers the data is loaded as you move the map and zoom in/out. In the case of OSM data the OpenStreetCam layer data is loaded only for the downloaded area.

The plugin saves the open/closed state of the layer. So if you delete the layer then at the next JOSM session the OpenStreetCam layer will not be loaded by default. A previously deleted OpenStreetCam layer can be activated again from the Imagery menu.

OpenStreetCam panel
In the OpenStreetCam panel you can interact with the currently selected image.

The panel along with the image displays basic information such as: OSM username and date of creation.

The panel also has a number of action buttons on the bottom. These are for filtering, next/previous image loading, centering the map, opening image web page and giving feedback. Image related actions are enabled only when the image is showing in the panel.

These features will be discussed in the next sections.

The plugin saves the open/closed state of the panel. So if you delete the panel then at the next JOSM session the OpenStreetCam panel will not be opened by default. If you don’t see the panel you should be able to open it by selecting the OpenStreetCam icon from the left side panel.

Image filtering

The displayed data can be filtered based on the creation time and JOSM user. In order to view only your uploaded images, you need to authenticate in JOSM using OAuth login.

 

By default no filter is set, custom filters can be removed by clicking the Clear button.

Visualizing an image and corresponding track
Individual images can be visualized by clicking on the image icon displayed on the map. The corresponding image is loaded in the OpenStreetCam panel and the corresponding track is displayed on the map.

An OpenStreetCam track is illustrated with a blue directed line. Images belonging to the selected track are illustrated with opaque icons; while other images along the track are illustrated with transparent icons.

Image zoom in/out

The displayed image can be zoomed in and out using the mouse wheel. In an already zoomed in image details can be observed by moving the image left, right, up and down.

Next/Previous image

You can navigate between the previous and next image of a track either from the OpenStreetCam panel by clicking on the Next/Previous button or by pressing Alt-Left arrow/Alt-Right arrow.

If the next or previous image is not visible in the current view, the map is moved automatically and images near the track are downloaded.

Center map to selected image

The map can be re-centered to the selected image location by clicking on the “Location” button from the OpenStreetCam panel. This feature is useful when the map was moved and the selected image location is not visible on the map.

Image web page

The selected image web page can be opened by clicking on the “Globe” button from the OpenStreetCam panel.

Upcoming features

We are working on improving our JOSM plugin and plan to add new exciting features. In the near future we plan to:

  • improve image loading speed by adding caching mechanism
  • allow the user to select easily nearby images to an already selected image
  • improve the map view and suggest street view coverage by displaying OSM ways instead of individual images. We will implement something similar as in the case of the web and mobile applications.

Source code

The source code for the plugin can be found on GitHub .

 

Feedback

Ideas, suggestions and bug reports can be submitted either to the plugin’s GitHub issue page or to the Feedback forum. Other mapper’s idea can be voted there.

We take a look at all incoming ideas, so be sure your input is heard and very much appreciated!

 

Have fun adding missing map features using OpenStreetCam images.

Facebooktwittergoogle_plus

Enhancing OSM Maps using Machine Learning & Big Data

One of our main goals here at Telenav is to constantly improve the maps we are using in our applications and services Having very detailed and accurate maps is of fundamental importance if we want to build high-quality and precise routing applications, ADAS systems or self-driving guidance software. In this post we’re going to talk about how we leveraged our massive datasets of anonymized GPS (probe) data in order to enhance the OSM maps, more specifically how we were able to detect missing roundabouts throughout the world.

The Task

The problem at hand is as follows: Given a dataset of GPS probe data and the current OSM geometries, could we identify missing roundabouts? More precisely, we are searching for geometries that lack a specific tag ( junction = roundabout) identifying them as such.

A relevant case would be the one below, where the geometry clearly defines a roundabout, but that specific tag is missing from the OSM map.

1484816109 1484816464

What we decided to do is to analyze car movement patterns and use this analysis to make inferences about the underlying map topology. In order to solve this not-so-trivial problem we decided to harness the power of Machine Learning. We did this because we are aware of the huge recent developments in this field and of the powers of a well-designed Machine Learning algorithm when combined with huge datasets. 

The Solution

The intuition about why this approach is preferable is obvious when analyzing the available data and how different traffic patterns are when we are in the context of a roundabout compared to the context of a normal intersection. What we have achieved is to teach the algorithm to associate the circular traffic movements having a “hole” in the middle with a roundabout and to associate the evenly spread movements with a normal intersection.

1484816471    1484816475

After successfully developing this “smart” detection algorithm, we have selected from the world map approximately 117 000 potential points, where a roundabout would be likely to exist based on some predetermined criteria. Of course, these are far too many to manually check, so the automated solution is the only one suitable for this job.

The Results

After running those points through the Machine Learning algorithm, it has detected around 9000 missing roundabouts in Europe and North America, as those are the areas for which we have GPS probe data. The massive size of these results which translate to substantial improvements of the OSM map is obvious when visualizing them.

1484816482

1484816492

1484816499

After a quick series of manual testing of a batch of results, we have discovered that the predictions are between 82% – 86% correct, depending on the level of confidence, which proves the efficiency of the Machine Learning oriented solution to this difficult task.

What’s next

In the near future, we plan to release this data to be validated by the OSM community through the MapRoulette platform and we are eager to see the feedback we get. Having acquired even more knowledge in this field, we are now ready to tackle more difficult problems using more advanced Machine Learning and Deep Learning algorithms. This will surely enable us to improve the OSM maps even more.

PS

We have uploaded all our predictions here in CSV format for those of you who are interested in playing with the data.

 

 

Facebooktwittergoogle_plus

Usage tips on the new Improve OSM

In this post we will take a detailed look at the new improveosm.org web site. We recently completely overhauled the application. It is now based on the OSM iD editor. We will walk you through the functionality and give some pointers.

Zoom and layer activation

Just by entering the web application, you already have our Improve OSM layer selected, and also active. A low zoom level displays a heat-map the content of which is modifiable via the left side filtering options. Note the color coded dots, corresponding to the heat-map’s circles.

improve-osm-heatmap

A higher zoom level, meaning a zoom level over 15, will show each individual Improve OSM item. Turn Restrictions are grouped in clusters, if some items represent rules for the exact same intersection. You’ll also notice that different types of Missing Roads have their specific color on the map and are also marked with the respective color in the filter panel. Filtering has the same usage as in the heat-map.improve-osm-transparent-tr

We talked about the Improve OSM layer being active by default, but what exactly does that mean? It means you can use the filters from the side panel and you can also interact with any Improve OSM item on the map. While inactive, the filter panel becomes minimized and any Improve OSM item becomes see-through. You can’t interact with the layer in this inactive state, but you can and will want to interact with all the iD’s map items and editing mechanisms. The SPACE key switches between the active and the inactive states. You can also use the toggle button found in the side panel’s header.

Item selection and status

When the Improve OSM layer is active any item can be selected. After selection, options appear in the side panel allowing you to change the status and add a comment.

You can have a multiple selection by keeping the CTRL key pressed and selecting items with the mouse. Only for the Missing Road items, you can batch select neighboring tiles. For this, keep the SHIFT key pressed and click one of the tiles in the batch. All neighboring tiles in that batch will be selected.

improve-osm-selections-1

improve-osm-selections-2

For Turn Restriction clusters, selecting one will automatically select a single Turn Restriction from the cluster. You can switch to another item in that cluster by selecting it from the list appearing in the side panel.

improve-osm-selections-3

Clicking anywhere on the map area not containing an Improve OSM item, will deselect all selected items.

For Turn Restrictions, you can select one to see its describing arrows and pass numbers, and then have the entire Improve OSM layer inactivated. This way, you’ll still see the describing arrows while being able to interact with the editing iD tools.

Item types

Turn Restrictions – they mark an intersection where a new, unmapped, turn restriction is in place. The green arrow marks the street vehicles came from while entering an intersection, while the red one marks a street used by none or very few of those vehicles to exit the intersection. It’s assumed that the very low percent of passes on the red segment, indicates a strong possibility of a turn restriction being in place. Passes on the red segment are assumed to be traffic rules violations.

Missing Roads – they mark a portion of the map (a tile) that contains GPS output from vehicles, in places no road is mapped. If they are numerous and they represent a clear trail, it’s assumed a missing road is there.

One Ways – they mark a road that is not mapped as a one way street, for which traffic data suggests with high confidence it actually is a one way street.

A typical usage flow

1. Identify an Improve OSM marker (item) and decide if iD editing is required

improve-osm-flow-set-1

2. Press SPACE to inactivate the Improve OSM layer and use the iD’s tools

improve-osm-flow-set-2

3. Make the wanted edit in iD

improve-osm-flow-set-3

4. Press SPACE agai, select the resolved Improve OSM item and change the item’s status to ‘SOLVED’

improve-osm-flow-set-4

We hope you find the new improveosm.org web site useful and are looking forward to hearing your feedback!

Facebooktwittergoogle_plus

A glimpse into the future of Mapmaking with OSM

We have over the last 12 months starting to look extensively in how we can leverage AI / Deep Learning to help improve OpenStreetMap and today we want to provide a few details about how we envision the future of making maps and also share more on what we are already doing. We see the emergence of self-driving vehicles as a game-changer and one key requirement for those vehicles are accurate and up-to-date maps. Currently commercial map providers map every region around every 12-24 months – in a costly process with a high precision and high cost vehicle, our goal was to achieve maps that are updated on a minutely basis and with key streets covered at least once every day. This is the goal we set out to solve with OSM in supporting to make it ready for this use-case.
Using OSM for Navigation Maps
At Telenav (and before at skobbler) I’ve been actively involved in OSM for almost 10 years now and it is truly unbelievable how OSM has grown massively in that period from a map that was used mostly by passionate enthusiasts to a map that is used by 100s of millions of users and big companies such as Toyota, Tripadvisor or Apple to just name a few to power their consumer products. Despite this success we have still seen that for navigation maps many additional attributes are needed that are not that well covered in OSM such as Signposts, Speed limits, Turn restricitons or Lane Information is needed to provide the best possible guidance.
 space
rsz_speed-na-1

Speed limit coverage

rsz_1tr-coverage
Turn restriction coverage
 space
rsz_tr-na-1
 Turn restriction coverage United States
 space
What we have done especially to close the turn restriction gap is to use (anonymised) GPS probe data from our millions of customers and from partners like Inrix to detect where there are likely turn restrictions based on turn behaviour. This data is then shared with the community via ImproveOSM and also for the most likely cases we put a high penalty on turns for our customers so they avoid those manoeuvres if possible. This way we have been able to detect 139,181 turn restrictions and increased  coverage in a meaningfull way.
Next step: Higher accuracy with Computer Vision
With Speed Limits, Lanes and Sign Posts it is significantly more tricky as it is not possible to identify those purely from GPS probe data. This is the reason why we started our OpenStreetView project to capture those images as there was no truly open project for Streetlevel Imagery that we could use (when we approached Mapillary they asked for hundreds of thousands of dollars in license fees – which was not an option for us).
In parallel to the OpenStreetView projects we have invested a lot in Computer Vision algorithms and established a cooperation with the Technical University in Cluj to get their over 15 years in the field. Our goal was to use computer vision to automatically build maps based on this images.
In the last year we made very significant progress and now we are able to detect Speed Limits, Turn Signs and Signposts (incl. OCR the text in those signs). Those detections when made will be reviewed by our editors and added directly into OSM.
<Slideshow with our computervision images for detecting turn signs, OCR, speed limits>
   input-picture-1
Input picture
space
   panel-detection-2
Panel deetection
space
glyph-segmentation
Glyph segmentation
space
character-grouping-into-words
Character grouping into words
space
ocr
OCR and classification results
space
We have build a map editor that allows us internally to review those changes and add them with our team of 20+ mappers to OSM.
We have by now added 19,798 map features (turn-restrictions,one-ways, signs) to OSM using this tool, and are adding every week hundreds of new turn restrictions and other signs to the map to make it better.
<MAP EDITOR TOOL SCREENSHOTS>
map-editor-1-cropped
 Map editor tool
space
map-editor-2
 Map editor tool
Advanced level: Create High Accuracy maps (ADAS / HD maps)
The next level for this challenge was to create the high accuracy maps needed by self driving cars and for ADAS (Advanced Driver Assistance System) applications. Those maps need accuracy < 2m which typically OSM doesn’t provide consistently and which is a big challenge to achieve purely based on GPS probes as we learned through a lot of trial and error. We looked into how we can achieve better accuracy and our natural choice was to leverage car data that is available to achieve higher accuracy.  Therefore we integrated our OpenStreetView application via an OBD2 port (which is available on every car manufactured in the last ~20 years) to integrate our phone based data with data coming directly from the car (such as speed, or on some models even with steering wheel angle available via OpenXC). With this we have been able to achieve an accuracy which is 5-10x higher than purely achievable by Phone based GPS and with several passes on one road we can create truly high accuracy maps.P ENHANCEMENTS FROM HARALD>
bildschirmfoto-2016-11-21-um-14-02-43
Trip Enhancement
Our vision of the future of map making:
We believe if enough consumers help recording the necessary images via OpenStreetView maps can be created in near real-time at an unprecedented accuracy. This would be a major enabler for self-driving cars and uptodate navigation systems. In order to make that possible we are also in early stages working with several car manufacturers to use the data from their on-board cameras in the future for those detections and hopefully this way we can use millions of cars from our OEM partners in the future with this technology to enhance maps and share this data with the OSM community to create even higher quality maps than today.
We will over the next few weeks go in this blog deeper into our individual modules that we built for making this future happen and looking also forward to feedback from the community.
<TEAMPICTURE OF TELENAV OSM TEAM>
thumb_img_8533_1024
team-pic-2
space
map-2
space
space
rsz_img_8680
foto_miriam frederic steve-coast
OpenStreetView team
Facebooktwittergoogle_plus