In my previous post, I presented interesting trips patterns of NYC taxis. Here, I focus my analysis of Brooklyn. Brooklyn is a very popular place of interest for both tourists and local people to eat, drink, and have fun. I usually take subway train L or G to visit Brooklyn. However, L train is to be shut down in 2019 for 1.5 years [1], which may severely affect Brooklyn’s business. Taxi trips, in particular, shared taxi trips, may become the mainstream transportation in Brooklyn.

To understand possible shared trip, I grouped pickup and dropoff locations by zipcode, neighborhood, and county. In order to do this, I first converted coordinates to address using Geocoder API for green taxi. Since the API is slow, I used single days (Wednesday, Saturday, Sunday) to represent weekday and weekends. I also included Uber data in this analysis, which include the pickup neighborhood information. Check my first post of data wrangling here.

Trip within Brooklyn

I first looked at trips starting and ending within Brooklyn. Such trips are defined as local trip and they usually have short distances of 2 miles and travel time 10 minutes.

Weekday (Wednesday) pickup in Brooklyn peaked at 9 pm. Saturday pickup peaked at midnight, starting to increase in late afternoon, and lasted to Sunday morning. Sunday however, had a lower pickup late night (lower than Wednesday).

Where did taxi pick up and drop off passengers?

 

Animated pickup map by hour

Saturday Pickup in Brooklyn

 

 

The spatial temporal pattern revealed that evening and late weekend night were the busiest time for certain Brooklyn areas.

Top 10 zip codes for pickups are 11231, 11205, 11216, 11222, 11238, 11215, 11249, 11217, 11211, 11201

Top 10 zip codes for drop off are 11201, 11211, 11217, 11249, 11215, 11238, 11222, 11216, 11205, 11231

Top Neighborhoods

(image url https://en.wikipedia.org/wiki/File:Brooklyn_neighborhoods_map.png)

When I grouped trips by neighborhood names, the popular spots became familiar.  Williamsburg, Greenpoint, Park Slope, Clinton Hill were hot pickup and drop off spots. In particular, Williamsburg led the 2nd popular spot by 100%.

Uber is also popular in these neighborhoods

I looked at Uber’s pick up records on the same days.

Similar to Green Taxi, Uber trip in Brooklyn also had a morning peak and a evening peak on Weekday. Yet trips droped in the middle of the day. On Saturday (Friday) overnight and Sunday overnight, there were more trips, similar as Green taxi.

Top neighborhoods of Uber, again, include Williamsburg, Park Slope, Greenpoint.

Popular Routes

Now that we know most trips start and end at similar hot spots,  I then investigated popular routes in Brooklyn. If some routes are very popular, it may be possible to redesign bus/shuttle routes. Also, if passengers share the same routes, they may also share taxi rides, which only only help passengers to save money, but also increase the efficiency of taxi rides.

I started with Green taxi Saturday data and grouped trips by pickup and dropoff zip codes.

sat_route_zipcode = sat_bk_bk.groupby(['pickup_zipcode','dropoff_zipcode'],as_index=False).count()[['pickup_zipcode','dropoff_zipcode','pickup_datetime']]
sat_bk_bk_distance = sat_bk_bk.groupby(['pickup_zipcode','dropoff_zipcode'],as_index=False).mean()[['pickup_zipcode','dropoff_zipcode','trip_distance']]
sat_routes = sat_route_zipcode.merge(sat_bk_bk_distance)
sat_routes.sort_values('pickup_datetime',ascending=False,inplace=True)
sat_routes.rename(columns = {'pickup_datetime':'trip number'}, inplace=True)
sat_routes

	pickup_zipcode	dropoff_zipcode	trip number	trip_distance
1	11201	11201	542	0.989410
254	11211	11211	523	0.934589
265	11211	11222	517	1.243926
937	11249	11211	415	1.054024
540	11222	11211	353	1.300425
249	11211	11206	326	1.619724
948	11249	11222	326	1.163926
29	11201	11231	322	1.530280
16	11201	11217	320	1.289594
352	11215	11215	316	1.091582
279	11211	11249	286	1.002483
423	11217	11215	248	1.354758
962	11249	11249	234	0.743547
410	11217	11201	223	1.339327
4	11201	11205	221	1.522489
14	11201	11215	212	2.279104
354	11215	11217	197	1.170964
549	11222	11222	196	0.937908
264	11211	11221	194	2.662938
36	11201	11238	192	2.123646
563	11222	11249	187	1.270321
445	11217	11238	178	1.214775
932	11249	11206	176	2.181420
719	11231	11231	166	1.214699
10	11201	11211	165	3.591879
694	11231	11201	160	1.537688
246	11211	11201	156	3.810256
107	11206	11211	153	1.360523
425	11217	11217	144	0.840000
64	11205	11201	142	1.552254
...	...	...	...	...
486	11220	11214	1	3.120000
485	11220	11211	1	9.320000
482	11220	11205	1	8.690000
480	11220	11201	1	4.920000
478	11219	11230	1	3.300000
476	11219	11218	1	4.300000
554	11222	11228	1	11.690000
566	11223	11214	1	1.100000
625	11225	11234	1	4.030000
567	11223	11219	1	1.930000
618	11225	11224	1	2.050000
615	11225	11220	1	4.470000
606	11225	11209	1	7.450000
599	11224	11239	1	9.500000
597	11224	11236	1	8.080000
592	11224	11230	1	4.020000
585	11224	11221	1	3.430000
584	11224	11220	1	5.740000
582	11224	11212	1	10.200000
581	11224	11211	1	15.680000
580	11224	11210	1	6.100000
578	11224	11207	1	10.640000
577	11224	11205	1	15.100000
576	11224	11204	1	4.090000
575	11224	11201	1	12.580000
573	11223	11232	1	8.400000
572	11223	11230	1	2.050000
570	11223	11228	1	3.870000
569	11223	11226	1	4.350000
976	11417	11212	1	1.600000

From the list of popular routes above, I noticed that these trips had same or similar pick up and drop off zip codes, and short trip distance < 2 miles.

I defined shareable trips as trips with pickup time within 5 minutes and with same pickup and dropoff zipcodes. If there are more than 1 trip, number of shareable trips = total passenger number / 6, while 6 is the capability of each taxi. Otherwise, the number of non-shareable trip is 1, for each route and time window. Then for each pickup zipcode, I count the total number of shareable trips and non-shareable trips. Shared ride efficiency for each zipcode  = #shareable trips / (#shareable+#non-shareable trips). Hourly ride efficiency is calculated as the mean efficiency across all zip codes.

1 Group by pickup and dropoff zipcode (namely route), and count total trip number, total passenger number, aggr trip number, and non aggr trip number. If total trip number of a certain route is 1, aggr is 0. Otherwise, divided the total number of passengers by 6 given that 6 is the capacity of a taxi. Non aggr trip is 0 when there is aggr trip, otherwise, it is 1.

def aggr_trip(df):
    df_gp = df.groupby(['pickup_zipcode','dropoff_zipcode'],as_index=False)
    df_trip_num = df_gp.count()[['pickup_zipcode','dropoff_zipcode','pickup_datetime']]
    df_passenger_num = df_gp.sum()[['pickup_zipcode','dropoff_zipcode','passenger_count']]
    df_eff = df_trip_num.merge(df_passenger_num)
    df_eff.rename(columns = {'pickup_datetime':'trip number'}, inplace=True)
    df_eff['aggr trip number'] = np.where(df_eff['trip number'] == 1,0,np.ceil(1.*df_eff['passenger_count']/6))
    df_eff['non aggr trip number'] = np.where(df_eff['trip number'] == 1,1,0)
    return df_eff

2.Calculate shared ride percentage across all areas

def aggr_eff(df):
    df_eff = aggr_trip(df)
    df2 = df_eff.groupby('pickup_zipcode',as_index=False).sum()[['pickup_zipcode','passenger_count','aggr trip number','non aggr trip number']]
    df2['aggr eff'] = 1.*df2['aggr trip number'] / (df2['aggr trip number'] + df2['non aggr trip number'] ) 
    return df2

3 Average across all routes

def aggr_eff_all(df):
    if df.empty:
        return 0
    else:
        df2 = aggr_eff(df)
        return df2['aggr eff'].mean()

4. ..

Overall, I defined hourly shared ride efficiency.

From my calculation, I reported that over 15% of trips on Saturday after 4 pm to Sunday 4 am are shareable. Over 10% of trips on Wednesday morning during rush hour are shareable.

Using similar analysis used above, I also analyzed trips between Brooklyn and Manhattan.

Total Hourly Trip and Shared ride efficiency

From my definition and calculation of hourly shared ride efficiency, it seems that shared ride efficiency should be correlated to total number of trips. The more trips there are, the more likely shared routes exist.

I plotted hourly shared ride efficiency and total hourly trip number of each day, and find that in general, share ride efficiency changes as total hourly trip number changes. There are several peaks in hourly share ride efficiency (red). Sat 1 am, 2 am, 8 pm; Sun 1-2 am (prolonged peak at midnight) ; Wednesday 3 am and 9 am. These peaks may suggest that at certain time, passengers are more likely go to similar direction.

There are fewer trips from Brooklyn to Manhattan compared with trips within Brooklyn and the shared ride efficiency is quite low on Wednesday.  On weekend, shared ride efficiency is above 0.06 only after 7 pm Sat to early Sunday. There is a peak on Sunday afternoon to Manhattan.

 

Linear Correlated?

To understand the correlation between total hourly trip and hourly shared efficiency, I first calculated the pearson correlation coefficient from 3 days (sat,sun,wed), 2 direcition (bk-bk, bk-man), and reported  0.899934, suggesting they are quite highly correlated. Again we can see there are fewer number of Brooklyn to Manhattan trips.

From the above analysis, I realize hourly shared ride efficiency is related to day of the week, hour of the day, destination (bk or man), as well as the total hourly trip. I then built a linear regression model to fit the data and try to predict a hourly shared ride efficiency of any given time.

def add_field(df,col1,col2):
    df['to'] = col1
    df['day'] = col2

# use numerical data to encode date and destination
add_field(sat_bk_bk_df,0,1)
add_field(sun_bk_bk_df,0,2)
add_field(wed_bk_bk_df,0,3)
add_field(sat_bk_man_df,1,1)
add_field(sun_bk_man_df,1,2)
add_field(wed_bk_man_df,1,3)

Sample size: 144  ( 3 days, 24 hour a day, 2 destination)

Features are ‘hour’, ‘to’, ‘day’

The label is  ‘hourly_aggr_efficiency’

	hourly_aggr_efficiency	total_hourly_trip	hour	to	day
0	0.183372	1694	0	0	1
1	0.192233	1545	1	0	1
2	0.134911	1310	2	0	1
3	0.149407	1045	3	0	1
…	…	…	…	…	…

from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3, random_state=42)
reg = linear_model.LinearRegression()
reg.fit(X_train,y_train)

Using reg.score(X_test,y_test), I reported 0.82 R^2 score, suggesting a good fit of the data.

y_pred = cross_val_predict(reg, X_test, y_test,cv=5)

fig, ax = plt.subplots()
ax.scatter(y_test, y_pred,alpha=0.5)
ax.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'k--', lw=4)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()

Last but not the least, I used cross validation to show the comparison of predicted value v.s. actual test value. As we can see, at lower shared ride efficiency, the predicted value seems to overestimate while as shared ride efficiency increase, the model seems to underestimate the value. Overall, the regression model provides us a rough idea about the shareability of Broolyn Trips.

References:

[1] L train service between Brooklyn and Manhattan to be shut down for 18 months starting in 2019, http://www.nydailynews.com/new-york/train-shut-18-months-starting-2019-article-1.2725118