Job Requisition Number: 22242. This position provides data engineering and architecture services for the UC Berkeley Partners for Transportation TechnologyÃs Connected Corridors (CC) program. The data engineer will support the development of real time traffic management systems as well as research for transportation solutions involving prediction and control of freeway and arterial traffic. This individual will be part of a development team that takes the results of the research and initial code, and develops a system that implements that research in a first generation software suitable for demonstrating the estimation, prediction, and control of freeway and arterial traffic in a Los Angeles transportation corridor. In addition, the development team supports the research with data capture, storage, and retrieval; research frameworks and technical support. The ability to identify and implement large scale data solutions that manage a variety of data structures, data volumes and data update frequencies is critical to success in this role.
This position involves using advanced software and system design skills in a highly distributed environment using SQL (Postgres) and NoSQL(Cassandra and MongoDB) databases, cluster data processing technologies such as Spark, Java development, Java frameworks (Spring, Hibernate), REST services, Amazon Web Services (primarily EC2) and JMS (ActiveMQ). The target system involves high performance, high volume data capture and analysis and real time data processing. You will apply your skills to development of advanced applied research, production ready software systems that will be used to manage traffic within a highly complex urban environment (Los Angeles). The position involves data systems design and architecture, requirements development, data streaming solutions design and implementation, programming, testing, implementing, and maintaining throughout the entire software application lifecycle. The incumbent provides technical leadership and acts as a resource to other members of the team developing using data engineering techniques and practices.
The development team and its members must manage constantly changing priorities, shifting deadlines, a complex set of relationships between research and development teams, and significant technical challenges. The team works within an Agile (SCRUM) environment and is a high performing, focused development group.
This is a hands on position that requires a high level of both design and implementation skills and an ability to code in Java.
25% Responsible for designing data components and implementing results of research. Develops designs for critical data components including data persistence, data processing pipelines, real time and batch processing, data transformations, and map/reduce strategies for a variety of data elements, formats, sources, and targets. Is an expert at design and implementation, and often performs complex data development and engineering functions at a professional level. Knowledgeable in real time distributed systems handling high volume data processing and complex data models.
25% Initiates, designs and uses complex SQL and NoSQL databases in research and production environments. Assists in the database design and implementation, and designs and implements complex data access layers for functional, high performance, and flexible system development and operation. Works in both cloud and local environments.
5% Functions as data architect for complex engineering assignments. Represents the unit and/or Principal Investigator to whom they are responsible at meetings and conferences in matters related to engineering design considerations and financial requirements.
5% Provides estimates using time schedules and budgetary considerations.
5% Provides data architectural, design, and development guidance, providing critical decision making for key systems development efforts.
5% Initiates and recommends changes in development, maintenance and system standards. Is a key member team in defining development, design, and system standards, as well as effective implementation of those standards. Understands and implements as part of the larger team integration and build processes and release management.
5% May supervise in-house development of system or component prototypes.
5% Participates in team quality assurance and testing efforts. Develops and executes moderately complex test plans. Ensures the team applies industry standard testing methods, including unit, functional, performance, and integration testing.
5% Participates in team Agile development efforts and is a key member of developing schedules and project assignments.
5% Ensures work is completed in a timely and efficient manner.
5% Provides direction and guidance to other Development Engineers and Applications programmers.
5% Plans, organizes, and performs the work of a professional engineering team engaged in a specialized engineering phase of a training and/or research program.Ã¯ Advanced knowledge associated with the planning, development and construction of engineering systems involving both hardware and software. Ã¯ Advanced knowledge of specialized design work as well as the overall field. Ã¯ Strong creativity and problem solving abilities to apply principles, practices and procedures within discipline to complete difficult assignments. Ã¯ Requires advanced knowledge of data architectures, data pipelines, real time processing, streaming and programming in Python and Java. In particular, must have experience with large, distributed, enterprise level system development. Ã¯ Must understand all levels of system design including design patterns and their implementation at the software component, subsystem and enterprise system level. Ã¯ Requires knowledge of advanced system design and architecture concepts including the different types of NoSQL databases and various SQL databases (especially Postgres), data streaming technologies, networking, security, cloud computing, and data management. Ã¯ Requires advanced skills associated with data system design, modification and implementation. Must be comfortable with iterative design approaches. Ã¯ Has skills needed to develop and implement system implementation plans with respect to data hubs. Ã¯ Must be able to communicate technical information in a clear and concise manner. Ã¯ Must be self motivated, work independently or as part of a team, able to learn quickly, meet deadlines and demonstrate problem solving skills. Ã¯ Must have senior level skill and experience with Java and Python programming specifically as it relates to data pipeline construction and data interfaces for relational (Postgres) and NoSQL (Cassandra and MongoDB) databases. Must have skill and experience with data frameworks and tools such as Spark, Storm, AWS DynamoDB, or others. Ã¯ Must have experience working in distributed cloud environments, preferably AWS. Understanding and previous experience with AWS data management environment and tools is a significant plus. Ã¯ Must have technical team leadership skill and experience as a senior developer with small development teams of 5-15 members. Ã¯ Advanced communication skills, both written and verbal, to convey complex information in a clear and concise manner. Ã¯ Advanced interpersonal and presentation skills. Ã¯ Ability to work in a highly collaborative manner; readily assess complex challenges and barriers and recommend effective resolutions. Ã¯ Demonstrated ability to lead, motivate and influence others.
The University of California was chartered in 1868 and its flagship campus - envisioned as a "City of Learning" - was established at Berkeley, on San Francisco Bay. Today the world's premier public university and a wellspring of innovation, UC Berkeley occupies a 1,232 acre campus with a sylvan 178-acre central core. From this home its academic community makes key contributions to the economic and social well-being of the Bay Area, California, and the nation.