Passenger Environment Survey


Passenger Environment Survey is a quality-control type survey used by a number of transit agencies to rate and monitor performance. Generally a PES would include items such as cleanliness, litter, appearance, temperature, graffiti, trash cans, and other quality issues affecting customer perception and quality of the riding experience. There is usually a quality control criteria, and dedicated surveyors are used to gather data that summarize into periodic ratings.
In North America, the New York City Transit Authority, Bay Area Rapid Transit District, and Metropolitan Atlanta Rapid Transit Authority are three agencies currently using this type of surveys. However, many other transit agencies have similar surveys or quality assurance processes—but other names might be used to describe them.

Introduction

Quantitative measurements require clear and well-defined standards. The majority of PES
standards were developed from the same quality control criteria used by operating departments
to manage their employees. Quality control criteria are found in various agency documents.
Some are codified in rules and regulations, union contracts, standard operating procedures, and
other official documents; others appear in training materials, field manuals, and management
procedures distributed to field staff.
Use of actual operating standards is important: it lends PES a degree of official credibility, and
improves acceptance by management and the public alike. Use of a different criteria to
monitor performance would be unfair to both employees and management. Additionally, use of
operating standards ensures by default all field employees would be familiar with PES indicators,
and thus clearly understand the yardstick by which their performance is measured.

Data Collection

Because of PES's audit and merit assessment functions, the operator must ensure data collection
forces have no vested interest in survey results. Biases associated with using operating staff for
data collection have been observed in many organizations. Difference between traffic checker
ridership counts, conductor reported passenger counts, and ridership levels implied by fare
receipts often differ in commuter railroads and other transit organizations. To provide a neutral
data collection workforce, a cadre of specially trained surveyors can be used. Surveyors
report directly to a performance measurement unit in central administration, minimizing the
influence of operating departments.
Surveyors should be hired directly from outside through the usual HR process. Therefore, they start at the operating agency
with minimal pre-conceptions about operations. These procedures together ensure that PES truly
represents the perspective of customers – seeing the system as they see it.

Important Features

Several features are particularly important in this type of quality control survey:
1. Quantitative Measurements are Taken Whenever Possible: Surveyors are required to
count or measure quantities. Number of doors not operating is counted. Interior
temperature is measured with a digital thermometer. Quantitative data enables analysis
to be conducted after the fact, and reduce the risk of surveyor errors in judgment.
2. Auditing Information is Gathered Simultaneously: By collecting information such as
car numbers, boarding and alighting stations and times, an analyst can later establish
whether the surveyor was actually present at the correct location. The fleet numbers can
be cross-checked with block register information independently recorded by operations
personnel in signal towers. This is particularly critical when invariably a bad score
causes operating departments to challenge the validity of one surveyor's work. If
discrepancies are found, results are discarded and surveyor asked to explain how
mismatching car numbers appeared in their data. Additionally, spotchecks by surveyor
supervision helps to maintain data quality.
3. Failing Scores are Clearly Documented: Whenever a failing score is recorded, the
surveyor records the reason on the form. This demonstrates the surveyor understands
PES standards and their correct application. The data is also used to analyze root causes
of failing scores. The requirement to record specifics
allows for post-facto auditing of surveyor activity. Analysts regularly review the forms
to gauge surveyors' comprehension level and act proactively to prevent recurring errors,
by reinstructing surveyors as necessary.
4. Qualitative Measures Require Clearly Defined Rating Standards: Occasionally,
quantitative measure is difficult or impossible to collect. An ordinal 'rating' standard is
then used with clear definitions. Bottles and cans,
and food matters trigger an automatic failure rating.

Sampling for PESs

Statistically valid measurements require even and representative samples. Biases due to
sampling must be minimized, to ensure validity in PES's audit and merit assessment functions.
At the same time, sample size must be minimized for cost reasons. Striking a balance between a
statistically valid sample and minimum surveyor cost is not simple. Historical data analyses
suggest PES scores are correlated with time-of-day, location, and service route, even though
indicators themselves are independent. PES scores are also subject to clustering effects: data
collected sequentially on the same day may be adversely affected by common factors such as a
school condition, weather conditions, or special events. A pure random sample is an inefficient
method of collecting such data, but observations made sequentially cannot be treated as
independent observations.
Generally, a good statistical sample should fulfill three criteria:
1. Sample error margins should be appropriate for the intended use of the statistic.
2. Sample should be representative of the underlying population.
3. Data collection method should not violate assumptions used when designing the sample.
A representative sample weighs each sub-class of the population equally. Applying that
principle to PES requires analysts to determine a priori what factors might affect PES scores,
and stratify the sample based on buckets of these variables. However, PES scores are affected by
many factors. With few observations per reporting period and many scheduling constraints,
a true stratified sample is not possible. A heuristic can be used to minimize the
impacts of skews introduced by uneven sampling:
Passenger Environment Survey at the NYCTA was introduced by David L. Gunn, former president of the Authority.
PES was introduced in the First Quarter of 1983 and consisted of a handful of very simple
indicators designed to quantify how amenable and comfortable the transit environment was to
our customers. Although it is difficult to objectively assess the many conditions that relate to
passengers’ feelings and perceptions, the early indicators provided a systematic framework to
assess performance in these areas independently of anecdotal customer comments. In that sense
the early PES was a big step forward and an important recognition that NYCT cares deeply about
the environment that is presented everyday to our 7.3 million customers. The early PES
included 11 subway indicators, and 21 bus indicators.
Several indicators were added in 1995, expanding the scope of PES. Indicators relating to the station environment such as functioning turnstiles and escalator/elevators were added. A major revision occurred in 2000, when indicators specifically designed to measure the performance of cleaning crews were added. Quarterly reports were scaled back to semi-annual beginning in 2004. NYCT began monitoring the passenger environment aboard express buses in 2007. Additionally, the PES survey methodologies have been used on a variety of "pilot" policy initiatives designed to measure the impact of allocating additional production resources to cleaning functions. The PES approach has been such a success that NYCT regularly receives requests for details, nationally and internationally, from other transit properties wishing to model quality control programs after NYCT's PES.
Initially, PES data was tabulated manually, and reports were issued quarterly with substantial time lag. Although this enabled managers to act after the fact to improve performance, many months may pass before substantial performance improvement is achieved. With the advent of personal computers, NYCT began keeping PES data and reporting results using spreadsheet applications and databases. Internal reports are now issued much more frequently, allowing management to react to mediocre scores mid-quarter. Data mining capabilities allow special requests from top management to be handled quantitatively.
Today, NYCT routinely produces internal PES monthly reports as a snapshot to assist operating personnel in taking corrective actions. The PES database is being migrated to Transit's enterprise server. The data collection effort is being transitioned from pen and paper to handheld computers. The PES staff provides regular 'Roadshows' in field locations to improve operations managers’ understanding of PES ratings and how they might improve PES scores, thereby improving the customer experience. These PES Roadshows have become an important mechanism of information exchange, improving relationships between PES and operating staff, and allows PES staff to gain valuable understanding of operations processes.
New York City Transit's Passenger Environment Survey is a quantitative and scientific approach to measuring passenger perception of NYCT's services. The PES standards are consistent, well-defined, and clearly understood by operations personnel. The indicators are measured from a passenger perspective by a group of dedicated and impartial surveyors, and
subject to rigorous data quality control by trained statistical analysts. Internally, PES functions as a performance audit for field operations management. Externally, PES serves as a dispassionate and analytical 'scorecard' measure of passenger experience.
First designed in 1983, PES has evolved over time to include 68 indicators in four distinct categories measured in four different passenger environments. Nine survey forms and 25 surveyors produce semi-annual reports with statistical precision exceeding 95% ± 5%. NYCT has such confidence in these measures that PES results are considered in the promotion and merit
decision-making process. The rich and extensive dataset generated by these periodic sample measurements are used throughout NYCT to support quantitative management decision making.