FRIB Operators

If you're reading this then you probably just recently joined Ops, so welcome!

The role of the operator is varied and can be hard to briefly summarize, so this page will ideally un-briefly summarize what you can expect to do on many average days. This will also focus on the actions mostly taken by newer operators, so don't be surprised if some of the more experienced ones have a different work flow.

• The primary purpose of operators is to directly interface with the accelerator, it's our job to make sure the beam and greater machine are in a desirable state. Operators have many unique privileges when compared even to the physicists who work here, so we are often asked to change things to better suite what they need out of the accelerator.
• An operator always needs to be in the control room whenever beam is being sent, if all of us leave when beam is being sent it's a big safety issue, so we often operate under a 24 hour shift. This also means that we are the best choice to monitor things overnight while system experts are at home.
• When you consider these two points, you'll start to see why the role of the operator can be challenging, we have our fingers in many pies of many flavours, yet we only have limited contact with most of them. If the Lithium Stripper causes us issues and prevents us from running, we're trained to understand the basics of what could be happening, but it's a challenging piece of equipment that we only interact with in a superficial way. We need to know enough about how it impacts beam to fix smaller issues or troubleshoot when there's no clear reason for problems, but we also need to do this with many other systems.
• All of this to say, operators need to know a little about a lot of things. This can feel frustrating or overwhelming at times, especially when in a high pressure situation, but you'll pick up a lot of this with time and experience, whether it be a systems expert or another operator, never shy away from asking questions about what something is or how to do an action that was asked of you.

• The most basic and common task that we do is work with the Chopper. The Chopper is a very powerful electrostatic that can ramp very quickly, it redirects beam from going further down the beamline. It is the main way we control whether beam is being sent, despite the beam travelling a decent distance from the source, where they start off, to the Chopper, it's not being sent to any measurement device or experiment, so we wouldn't refer to this as “sending beam”.
• We control the Chopper with computers that have “ftc” written on their labels. We have 3 different networks, you're probably reading this on one that says “ofc” on it now. “ftc” machines can be used to control most of the accelerator, until you get out of ARIS, but don't worry about that now. Ignoring a lot of technical information about the computers, there are different pages that control or display different things. One page is “MPS Operator” and this is where we control the Chopper. There are large, clearly labelled buttons that allow us to “Stop Beam” or “Start Chopper”, there is some confusing language regarding the Chopper, normally when an Operator refers to the Chopper as “being on” or “starting” that actually means they're turning it off to allow beam to travel further down the beamline. “Stop Beam” then powers on the Chopper and “Start Chopper” actually turns the Chopper off and allows beam to continue along.
• The most basic thing we're responsible for is making sure that beam is being sent when it should be, and not being sent when it isn't. But it's also crucial to keep track of the beam's details. When you have the chance, mosey over to the large TV's hanging above the Control Room doors. This provides a great map of the accelerator and other parts of the beamline, but also some very important numbers. The one I find myself checking the most is Beam Power, which can be thought of as a measurement of how much beam is being sent. If the beam is being stopped by the Chopper then Beam Power will be zero. You can imagine that there are some pretty tight restraints on our Beam Power, and another one of our key responsibilities is keeping this at an acceptable figure.
• What is acceptable? Great question, our Operational Safety Envelope or OSE is a document that provides guidelines for what we're allowed to do. Violating the OSE is a serious matter, but exists so that we can self govern to some extent and avoid violating the Accelerator Safety Envelope, which is external and could get us in some pretty serious trouble. They are intimidating documents, but feel free to ask other Operators for assistance when referencing or reading through them.

Notes - In the second point, perhaps take out some of the details about the chopper. Reserve these for a good and thorough explanation in another section dedicated for specific equipment in the beam line. Replace with some description of how pages on Phoebus are displayed, and how to navigate them?

• Don't worry! The annunciator might be constantly reminding you that there's “30 Active Alarms”, but having some be active is normal and more steeped in office politics than I'd like to go into. You should not however ignore the annunciator.
• Firstly, you may have noticed that there are two voices for the annunciator. One is a male voice that speaks much more frequently and one is a female voice that speaks infrequently, but will most likely repeat herself a few times. Theses are separate alarm systems, we'll look at the male voice first.
• This is the Phoebus Alarm System, named after the software it lives in. Phoebus Alarms mainly come in three varieties, all being announced before the rest of the alarm. Those are “Minor”, “Major”, and “Invalid”(pronounced “In-va-lid”). You'll also see “Undefined” on rare occasion, but these often are apart of some larger bit of work going on. To explain these, let's imagine some measurement that gives us 20 of some unit. Many devices are being monitored and measured so this could stand in for many devices,I'm just using 20 as a simple number. There are generally 4 limits associated with any measurement, the HI, the LO, the HIHI, and the LOLO. The HI and LO and numbers that are above and below the expected measurement, so for our example the HI could be 23 and the LO could be 17. These are often set as warnings that the measurement is shifting and that you should look into them before they potentially get worse. After that the HIHI and LOLO values are more extreme in both direction, so HIHI could be 25 and LOLO could be 15. These will tell us that the situation has gotten worse and things could potentially be in a bad state. HI and LO alarms will trigger a Minor alarm and HIHI and LOLO will trigger a Major alarm. Try to keep an ear out for these kinds of alarms.
• Invalid alarms notify when a measurement is “disconnected”, there was some error in communication that are normally very short, but these should always be checked to make sure connection was reestablished.
• For response advice, check here.
• A common alarm you will hear is “MAJOR ALARM: MPS FAULT”, which adds to the ridiculous number of acronyms we use here, but this one is very important. What does it do? It stops beam with the Chopper. Why did it happen? Well that's a good question that leads us down a longer rabbit hole.

• The Machine Protection System is a system. It's purpose is to protect the machine. Sardonicism aside, it takes those measurements we discussed before and watches them. Some of those measurements can go into alarm without threatening the health of the machine, but many of these measurements can tell us that it is unsafe for the machine to try sending beam. NOTE: MPS only protects the machine, I'll talk about our Personnel Protection System soon enough.
• A few things happen when MPS finds a measurement that stops us from sending beam, firstly remember that it does not watch every measurement. Some measurements correspond to devices that don't directly impact beam and therefore aren't watched. Some measurements correspond to situations that do impact the beamline, but are localized to an area where we are not sending the beam. Some measurements do not live on the same system as the “ftc” network that controls the Chopper, so even if they do tell us that something is wrong and we should stop the beam ASAP, they cannot be read by MPS to stop the beam automatically.
• So let's say that we're sending beam and everything is working alright, suddenly “MAJOR ALARM: MPS FAULT”. Now, we're not sending beam, that's stopped by the Chopper. If you try to tell the Chopper to send beam, it won't listen. This is because the device for whichever measurement went bad is now Interlocked. Measurements can vary quite a bit, most have a certain “fuzz” of variation from noise or device fluctuations and to ensure that this doesn't cause the device to go in and out of alarm unceasingly, it may enter an Interlocked state. The device should be checked (quickly looking at it's other measurements is ok in many cases) before the Interlocked is reset. The Chopper may not be restarted until all devices have good measurements and are not Interlocked.
• One final note is that MPS does not watch any measurements when the Chopper is off. They are very closely linked and sometimes it's convenient to allow devices to be adjusted without causing 100 alarms, so this can be done at any time by stopping beam with the Chopper. Sometimes we would like to watch those measurements but not send beam, this can be done by blocking the beam with devices other than the Chopper, for now it's good that you know this but the details aren't urgent to know. Feel free to ask another Operator to satiate your curiosity (Hint: mention “1102” and they should know exactly what you're talking about.)
• Read more here.

• Voice Alarms are the other variety of alarms you'll hear on occasion, they have a female voice and will repeat until an Operator acknowledges them, They live on totally different software than the Phoebus Alarms and are generally more scary. The main differences are that these alarms can encompass personnel safety, such as low O2 levels in an area or excessive flammable gas in another. There is also a big folder of responses to these alarms behind the main desk labelled “VOICE ALARM RESPONSES”. If there is ever a voice alarm, I would recommend trying to find it in that folder if nobody else is looking at it.
• Similarly, there is a big yellow box to the side of the main desk, this is the PPS Rack. It's main purpose is to make sure that there are no people in areas that will contain beam. It works in a more complicated way, but that's its main function. Before sending beam, we need to make sure that the PPS Rack says it's ok first. If the PPS Rack doesn't think it's ok, then it yells real loud and it's the scariest sound in the Control Room. It looks at some measurements for devices that you can control on the “ftc” network, but it also looks at which areas are “secure”
• Securing an area means doing a Search and Evict which is a sweep of the area to ensure no one is in there. Operators do many Search and Evicts, but other trained individuals can also perform them. During the Search and Evict buttons are pressed throughout the area as it is searched for people, at the end all entrances into the area are closed and the PPS Rack will acknowledge it as secure. If there is any error in the Search and Evict or if the doors into the area are opened then the area will no longer be secured. This can range from an inconvenience of having to re-secure the area to the PPS Rack freaking out as it is possible that a person could have entered an area containing beam. This will violently shut off many devices and is very bad for them. Accidents happen, but ensure to read the Stack Lights and Signs outside of every door before opening it.
• Read more about Voice Alarms or PPS.

• I've written quite a bit here, feel free to skim or reference it at your leisure, but it's a verbose overview of what we do here. It should give you a basic idea of introductory responsibilities for an Operator. Hopefully this also worked well at introducing you to the job more than the machine, but knowing the machine is important so you can check out What's an Accelerator?.

• There are three different shifts you may be scheduled on. Day: 7:00 am - 3:00 pm, Afternoon: 3:00 pm - 11:00 pm, or Night: 11:00pm - 7:00 am. Eventually, you will begin doing weekend shifts. These consist of nine consecutive days on, including the weekend, which will be two 12 hour shifts. Before or after starting weekend shifts though, you will have three work days and the weekend off, which is a nice way to prepare or relax for these shifts.
• Each shift can vary in how they feel throughout the time you're here. Day shifts are typically quite busy, afternoons are busy in that first half usually and a bit calmer towards the end, and night shifts can be calm, but due to it being so late you'll need to be prepared to solve issues or call people early in the morning. All this being said though, shifts can vary depending on what is happening that day. Day shifts may be slow but afternoon could be packed, vice verse. The point of this is just to inform you of what to kind of expect when you are here during those first couple of weeks on shift. As said above though, don't worry to much about this as time goes on because you will eventually experience each shift yourself.