Managing SCADA During a Municipal Water/Wastewater Capital Project

Automation continues to play an increasing role in providing critical infrastructure including public water and wastewater facilities. Gone are the days of operators driving from site to site to turn valves and start or stop pumps as part of normal operations. Instead, we now use sophisticated supervisory control and data acquisition (SCADA) systems to automatically control and monitor the wide range of facilities that provide the drinking water and wastewater services we all depend on.

Building this critical infrastructure involves much planning, design and resources, not to mention many people with a wide variety of skillsets. This article provides an overview of the typical workflow involved with designing and building a municipal water/wastewater capital project, and the role that automation professionals play.

It’s a team effort

Good projects are always a team effort. No matter what methodology is used—traditional design-bid-build, design-build, design-bidbuild-operate, etc.—the common element is a wide variety of individuals and skillsets that need to come together to execute a successful project. Obviously, having a common goal, ensuring all parties work together, and ensuring the plant works—as well as ensuring everyone gets paid—are all of paramount importance.

The coordination of these efforts is usually undertaken by a group of project managers: One acting for the owner, one acting for the design team, and another acting as part of the construction team. It is the project managers who oversee the overall timeline and do the important tasks of controlling the schedule, scope, cost, and quality aspects of the project. Historically, most project managers at this level come from a civil engineering/ construction background. Civil engineers are generally not specialists, but typically will have a broad background that allows them to have a deep understanding of what is needed for each phase of the project and who will be needed to carry out the work. The project managers will then bring in various specialists as needed during the duration of the project. If a project involves SCADA in any way— which is pretty much a foregone conclusion these days—automation professionals will be needed throughout the project.

Scoping the project

Most municipal water/wastewater projects start with a study to confirm the need and timing for the upgrade (or new facility). The study is typically followed by securing funding, and then the development of a charter to define the project.

Once a project charter has been developed, the first task for the utility is to develop a detailed scope of what the project will include (and not include) and the intended plan for staging the work. To do this, a terms of reference (ToR) document is typically created for each of the major project teams. For a traditional designbid-build project, the ToR will provide a task-based overview of what aspects the design team will be handling. The design team will, in turn, develop the builder’s scope in the form of contract drawings and specifications.

Fast facts about SCADA

• In the municipal water/wastewater sector, automation systems are referred to as SCADA systems and typically include instrumentation, signal wiring, programmable logic controllers (PLCs), motor control centers (MCCs)/motor starters, actuated valves, the control network, servers, workstations and alarm callout systems.

• SCADA systems can be implemented in the municipal water/wastewater sector using a variety of technologies including PLCs with human-machine interface (HMI) software, distributed control systems (DCSs), Industrial Internet of Things (IIoT), and proprietary controllers.

• Once installed, SCADA equipment in the municipal water/wastewater sector is expected to have a service life of 20 to 30 years, which is considerably longer than many other industries.

From the design team’s perspective, the ToR document will outline the design goals and provide a list of the deliverables and services the owner is expecting. A typical ToR includes task descriptions such as collecting background information, developing a design brief, preliminary design, various detailed design stages (e.g., 50%, 75%, 90%), and creating a ready-for-construction set of drawings and specifications, as well as administering the construction contract. In effect, the ToR for the design team forms the basis for many of the stages of a typical municipal infrastructure project. A summary of typical project stages is listed in Figure 1.

Background stage

Once the design team has been selected, the next step is to conduct a detailed survey of the existing conditions and legacy systems that must be incorporated (or replaced) by the new design. If it is a legacy site, this includes gathering historical documentation about the facility. From a SCADA perspective, this should include, at a minimum, a set of up-to-date piping and instrumentation diagrams (P&IDs), site layouts, floor plans and electrical drawings.

Ideally, electrical drawings will include not only power distribution drawings, but also drawings for programmable logic controller (PLC) panels, controller panels, motor starters, field wiring, input/output (I/O) signals, control system networks and any other control system wiring. Some utilities will even pay a third-party engineering firm to make a set of as-found electrical drawings, P&IDs and floor layouts, so these can be provided to the main project’s design team as part of the background information.

Design brief

Once the design team has a good understanding of the project scope and background information, they will develop a short report that outlines their proposed design solution. This is typically called the “design brief,” which can range from a few pages to more than 100. The design brief usually includes conceptual drawings. Sometimes the design brief is referred to as the “conceptual design phase” or a “10% design."

From a SCADA perspective, the design brief must also include a short section about the proposed automation hardware and how it will work.

Preliminary design

After the design brief has been reviewed by the utility and relevant stakeholders, the next step is to carry out the preliminary design. This stage is often called the “30% design.” There is usually a considerable amount of design effort that must be spent at this stage, as this is when the overall project plan is fully developed and the important question of “Will it all work?” must be sorted out. The preliminary design stage should result in a preliminary design report that outlines the various features of the proposed design and the rationale behind them, and it should be accompanied by a set of preliminary drawings.

Detailed design phases

Once the preliminary design has been reviewed by the utility and feedback has been gathered, the next step is to proceed to detailed design. A commonly used progression of detailed design stages is 50%, 70%, 90% and construction ready. The decision of how many detailed design stages will be used and if supporting technical memos are to be developed will have been defined in the ToR at the start of the project.

At the end of each detailed design stage, an increasingly detailed package of drawings and specifications will be provided to the utility for review. At each design phase, the number of drawings and length of the specifications will increase as well. For example, at 50% design, the specifications section will usually only consist of a table of contents, whereas at 90%, it is not uncommon for a specifications section to consist of hundreds, if not thousands, of pages organized into numbered sections.

The SCADA team’s role during detailed design

From a SCADA perspective, the main goal during detailed design is to ensure that all the various aspects of the design have been properly coordinated with each other, and that the utility’s SCADA design standards are being followed. The overall process systems, and the SCADA system that controls and monitors them, will only be able to function effectively if all the various aspects of the design have been well designed, coordinated and sized properly for all operating modes.

Operating modes include startup, shutdown, online, offline, normal operation and abnormal situations. Thus, both the utility’s and the design team’s SCADA staff need to be able to review all the drawings and specs together, not just the SCADA-specific sections, so they can check the overall design coordination.

Common SCADA and process design issues

Common SCADA and process design issues that are often encountered on municipal water/ wastewater infrastructure projects include:

• Improperly sized flowmeters: Flowmeters should be sized based on the expected flow rate, not the pipe size leading to them. If a flowmeter that’s too large is installed, it can read unreliably or suffer from sludge/sediment buildup due to flow velocities being too low.

• Poor locations for instrument taps: Welldesigned tap points for instruments are accessible and are clearly shown in the drawings. Good practices include having full port shutoff valves on each port for isolation, having spare ports and locating tap points so they won’t be prone to fouling/sediment.

• Sample lines that are too long: Long sample lines, or sample lines that are poorly sized, can result in long dead times for the sample to pass from the sampling point to where the analyzer will see it.

• Chemical dosing lines that are too long: Long chemical dosing lines create the same problem as long sample lines. Long dosing lines can make it difficult (or impossible) for chemical dosage adjustments to affect the process as desired.

• Misplaced or missing air relief valves: Not having enough air relief valves installed in the high points of piping systems and on pump discharges can result in problems with trapped air in pipes, vapor locking and erratic flow behaviors.

• Inappropriate use of valve types: Butterfly valves should not be used as control valves, as they do not effectively control flow and if used as control valves, they can rapidly deteriorate due to cavitation.

• Improperly sized control valves: Improperly sized control valves will never work properly, no matter how sophisticated the software-based control scheme is. For example, control valves should be sized based on an appropriate valve coefficient (Cv) rather than just the line size.

• Missing pressure testing instructions in contract documents: When no guidance is provided in the contract documents to the construction team, mistakes can easily happen. This can include rupturing instrumentation and diaphragm seals, incomplete testing or missing test documentation.

• Missing SCADA and/or network details: Incomplete process control narratives, missing PLC hardware details or no clear instrumentation and controls (I&C) wiring guidelines are a recipe for delays and extra costs during construction.

These are only a sample of the many potential design issues that can happen when the detailed design of a municipal water/ wastewater facility is rushed by a design team. The important takeaway here is this: Use a team approach to ensure that enough QA/QC is done on the design package to catch potential problems at the design stage, when they are much cheaper to resolve, rather than during the construction stage. Often, when a problem is not fixed during design, it can cost 10 to 20 times more to resolve during construction.

Construction phase

Once the construction project has been awarded, a construction kick-off meeting will be held, followed by regular construction meetings between the utility, the design team and the construction team. The construction schedule will be reviewed and details relating to the construction will be coordinated as needed.

As part of the construction process, the contractor issues shop drawings that consist of specification sheets and other submissions for the various products the contractor proposes to install along with information about how they propose to install them. These will be reviewed by the design team and the utility as needed prior to installation.

The shop drawings must be reviewed by the various design disciplines that are impacted by them including the utility’s SCADA team. Most SCADA professionals will want to review the shop drawings for automation equipment, instrumentation and any other equipment that “has wires” attached to it, to ensure the automation aspects of the project are properly coordinated.

SCADA construction aspects

While the physical construction is taking place, a system integration team works on the offsite SCADA aspects of the project. This typically involves PLC panels, instrumentation and automation programming. Depending on how the project is structured, these three aspects may be handled by one team or may be split up among several teams that coordinate with each other.

A series of factory acceptance tests (FATs) will be held for the various pieces of automation/SCADA equipment and software. These are typically reviewed by the design team and the utility’s inhouse SCADA team prior to installation. Depending on the aspect, they will also be reviewed by the utility’s operations and maintenance team as well. As with any fabrication/development phase, it is best to resolve problems as early as possible to avoid costly rework later.

Commissioning

Commissioning is the step where the various pieces of installed equipment, software and support systems onsite must work together as a system for the first time. Best practice is for a detailed plan to be developed and followed for the commissioning stage as it will involve a lot of people to start up the various pieces of equipment and support systems. Most commissioning teams maintain a commissioning logbook and ensure that all tests are documented and signed off as the plant commissioning proceeds. Commissioning can be very time consuming so it is important that is it carefully planned out and coordinated.

Commissioning usually starts by individually checking each piece of equipment before attempting to run it together. These individual checks usually start with visual inspections, configuration checks, test runs and testing various auto-shutdown conditions. Only after individual checks are complete can the entire system be tested together.

Also, equipment, systems and subsystems should always be tested in manual mode before attempting to run them under automatic control. Commissioning can be very time consuming because all the possible operating modes, and issues must be carefully checked, and adjustments made as needed, for all aspects of the facility.

From a SCADA perspective, commissioning usually involves power-on checks, I/O wiring checks, loop checks and informal testing, followed by formalized site acceptance tests and site integration tests.

Performance test

After all the commissioning tests are complete, the facility enters a performance test period in which it is expected to run with few, if any, adjustments. It is not unusual to see a seven-, 14-, or even 21-day performance test run specified in the construction contract for a municipal water/wastewater facility.

Close-out

The successful completion of the performance test does not signal the end of the project. The design and construction teams must now document what has been built in a series of submittals into various project closure documents which will include operations and maintenance (O&M) manuals and as-built documentation.

A typical contractor-provided O&M manual will include the specifications and manuals for every component installed in the plant, along with copies of the associated approved shop drawings, and copies of the configuration settings and commissioning reports that were used for commissioning it.

The system integrator will provide an O&M and as-built package that consists of backups of any automation code and documentation on how the various automation systems work. Finally, as part of the ToR’s scope of work, the design team will provide an O&M submission for how the plant is intended to operate from an operations point of view, plus a full set of as-built drawings to reflect how the plant was built.

Final thoughts

Building critical infrastructure is a complex process with many moving parts. It is a team effort that requires personnel who have a wide variety of skillsets working together during all the various project phases for the result to be a successfully operating facility. Because of the prevalence of automation, automation professionals are an essential part of this team effort. By being involved at every step of the process, automation professionals continue to make the world a better place with automation when it comes to critical infrastructure.