GlMapBuffer takes longer and longer

I have a problem with glMapBuffer. I'm making a simple program and I wanted to implement Sprite rendering. Since doing 500 draw calls for 500 sprites is really slow, I wanted to use one big VBO and update it every frame with new data. Good thing is, that it is faster. Weird thing is, that it slowes down as program is running.

I profiled it and the source of the problem is that calling glMapbuffer takes longer and longer. At first it uses few percents of one tick of the program, but after like a minute its already 36%. The profiler(VS community) just points me to nvoglv32.DLL, but that's dead end to me.

I seriously have no idea how to fix it.

Here is code of Sprite renderer: http://ift.tt/1IBzOz1 (when I copied it here, it looked really weird) and header file: http://ift.tt/1Hmjrd4

The are other things, but you can ignore them, they don't look like they cause the problem.

I initialize it, then in the main loop I basicly do:

renderer.Begin();

for(data)
   renderer.Submit(data.pos,data.dims,data.col);

renderer.End();

renderer.Render();

Is this correct way to implement something like this? I mean, maybe I'm forgetting some important glCall somewhere.

I let it run for 15minutes, now just mapping the buffer takes 20ms: ( I also updated my drivers, but even that did not help.