These instructions are intended to help you start using the Endace DAG cards. They are not endorsed by Endace, and if you’re stuck then you should refer to their manuals or contact their technical support people (who are quite helpful). These instructions are based on the DAG7.5G4 PCI Express card.

Interface Modules (SFPs)

Endace only support the Finisar SFPs. Others do work, and I’ve used a Cisco 10/100/1000 SFP that was borrowed from the IT department.

The DAG cards do not support 100BASE-FX SFPs – the only fibre interfaces supported are 1000BASE-X. If you need to monitor 100Mb/s fibre links then you will need a media converter, but any timing measurements need to take into account the latency of the media converter (which are often two port switches).

Basic Capture

If all you want to do is capture frames then set the card up as receive only, fix the ports to the correct rate and you’re done. Most of the time auto-negotiation of duplex works fine, but things do get messy with taps and half duplex devices.

dagconfig default Resets the card to default values

dagconfig 100 rxonly mem=128:0 Sets all ports to 100Mb/s and allocates all the memory to a single receive stream

Now it is time to capture some frames. dagsnap does this for you, but a trap is that it can also store data that has arrived in the meantime. The dagbits command can be used to clean the buffer out first.

The verbose (-v) option of dagsnap shows the total accumulated data, how much of the buffer memory is used and the incoming data rate (in that order). The recording time is specified with the –s option and output file with the –o option. A 15 minute capture to example.erf would be:

dagbits –d0 –cv –S2 Runs for two seconds and clears the buffer

dagsnap –s 900 –v –o example.erf Runs for 900 seconds and saves into example.erf

Viewing Files

Wireshark is a very easy way of viewing ERF files since it supports them directly. Converting the ERF file to a PCAP file with dagconvert reduces the time stamp accuracy. If you capture from multiple ports at once all the frames end up in one file, but the receive interface is stored in the ERF header. Adding a custom column to Wireshark that has the custom field erf.flags.cap makes it very clear where the frames came from. Colour coding based on the interface also helps.

enter image description here

Filtered Capture

Sometimes there is more happening on the network than you care about. DSM filtering is a nice way of doing this, but you need to use DSM firmware. The factory firmware in the DAG7.5G4 is edag75g4pci_dsm_v2_2, and this supports DSM. The latest version is edag75g4pci_bfs_v2_4 and this doesn’t support DSM. If you have upgraded to a new version and want to enable DSM then run dagreset

A DSM filter file is an XML file with special tags. Check the Endace documentation (EDM04-07 dsm_loader User Guide) for details. I want to capture data only if the Ethertype is 0x88BA, so my filter file is:

<?xml version="1.0"?>
<dsm-config version="1.0">
    <!-- SV filter -->
    <filter>
        <name>filter0</name>
        <number>0</number>
        <ethernet>
           <ethertype hex="true">88BA</ethertype>
        </ethernet>
    </filter>
    <filter>
        <name>filter1</name>
        <number>1</number>
        <ethernet-vlan>
            <ethertype hex="true">88BA</ethertype>
        </ethernet-vlan>
    </filter>
    <partial>
        <name>partial0</name>
        <partial-component>filter0</partial-component>
        <partial-component>filter1</partial-component>
    </partial>
    <stream>
         <number>0</number>
         <stream-component>partial0</stream-component>
    </stream>
</dsm-config>

Two filters are needed to ensure frames that are tagged with 802.1Q are also captured. Filters can be set up for particular source or destination addresses too.

The capture process is just like a simple capture, but dsm_loader is run first

dsm_loader -f filter.xml dagbits -d0 -cv 
                 -S2 dagsnap -v -s 15 -o capture_file.erf

The only way of going back to the user/updated firmware is to reboot the computer.

Transmitting Data

The DAG7.5G4 is capable of transmitting and receiving data at the same time. There are some extra setup steps required though. These involve setting up the buffer memory and disabling receive for an interface or two. The data to be transmitted needs to be prepared properly.

The DAG card transmits ERF files, and this means it can transmit from multiple interfaces. The simplest way is to replay a captured file, but you can also convert a PCAP file to ERF and specify which interface should be used. A trap is that the ERF data needs to be aligned to 64 byte boundaries, and this is managed by dagconvert. I haven’t replayed ERF captures, so this example will take some synthetic PCAP data.

To convert input_file.pcap to output_file.erf, with interface 1 (the second port) to be used, run:

dagconvert -i input_file.pcap -o output_file.erf -T pcap:erf -A 64 -p 1

The maximum buffer memory is 128MB, and this needs to be shared with receiving too. The DAG card is very efficient, and even 1MB of receive buffer is enough to capture 100Mb/s data on two ports (total of 200Mb/s going to the disc). The maximum ERF file size I’ve reliably been able to transmit is 125Mb. There may be tricks to work your way through a larger file, but I haven’t had much luck.

To setup the card to receive on interface 0 and transmit on interface 1, both at 100Mb/s and with the transmitted data to follow the timestamps of the ERF file run:

dagconfig default
dagconfig enablea disableb disablec disabled
dagconfig 100 auto_neg rxtx mem=1:127 relative

To transmit and receive, you will need to console windows.

On the transmit one run

dagflood –f output_file.erf –v –r2

On the receive window do your normal capture, such as:

dagbits -d0 -cv -S2
dagsnap -v -s 15 -o capture_file.erf

It is a good idea to wait until the dagbits command has finished before running the dagflood command. I give the dagsnap and extra 10 seconds or so, I wait until it has started capturing some data and then start dagflood.

Complicated Setup

To capture data on interface 0 at 1000Mb/s, capture on interface 1 at 100Mb/s and transmit on interface 2 at 1000Mb/s the following commands would be used:

dagconfig -d0 relative rxtx mem=1:127 enablea enableb disablec disabled  slen=1600 varlen
dagconfig -1 1000 auto_neg
dagconfig -2 100 auto_neg
dagconfig -3 1000 auto_neg

In the receive window:

dsm_loader -f filter.xml
dagbits -d0 -cv -S2
dagsnap -v -s 15 -o capture_file.erf

In the transmit window:

dagflood –f output_file.erf –v –r2

– Runs for two seconds and clears the buffer

These designators are taken from ACMA radio registrations and manufacturers' data sheets.

Analogue voice

  • 3K00J3E SSB Voice
  • 6K00A3E DSB AM Voice (Aviation, Marine)
  • 10K1F3E FM Voice, 12.5 kHz bandwidth
  • 11K0F3E FM Voice, 12.5 kHz bandwidth
  • 14K0F3E FM Voice, 20 kHz bandwidth
  • 16K0F3E FM Voice, 25 kHz bandwidth (4 kHz deviation)

Digital voice

  • 3K00J2D Codan Envoy LRDR Digital Voice
  • 4K00F1E NXDN (6.25 kHz) Digital Voice
  • 5K76G1E P25 Phase II CQPSK (6.25 kHz slot)
  • 7K60FXE DMR Digital Voice (TDMA 2 slot)
  • 8K10F1E P25 Phase I C4FM Voice (12.5 kHz slot)
  • 8K10F1W P25 Phase II H-CPM Voice
  • 8K30F1E NXDN/IDAS (12.5 kHz) Digital Voice

Mixed content

  • 6K25W7W D-STAR Voice+Data (also 6K25F7W)
  • 7K60FXW DMR Digital Voice+Data (TDMA 2 slot)
  • 10K1F9W P25 Voice+Data
  • 25K0D7W TETRA Voice+Data

Now that 2016 has come to an end I thought that it would be interesting to look at trends and changes in the way electricity was generated around the National Electricity Market. There have been a few significant events in the past twelve months in the electricity sector.

If you have suggestions or requests for other visualisations please let me know (via the contact page). I have 5-min SCADA data for each generator, so there is no shortage of information available.

New South Wales

  • Mostly coal and interconnector imports from Queensland (black coal) and Victoria (brown coal).
  • New solar PV stations are a good thing, but their output is lost in the noise compared to coal and interconnector imports.

Queensland

  • Interconnector to NSW flows south most of the time.
  • Some renewables, but insignificant compared to the coal and gas.
  • Does not show the rooftop PV contribution to load---this is the nett generation required.

South Australia

  • Coal-fired generation ceased in May. Gas generation (thermal and gas turbine) is now the only synchronous generation in service.
  • Wind power is a significant part of the state's energy mix.
  • The interconnectors between SA and Victoria generally flow west, increasing the carbon intensity.

Tasmania

  • Significant gas generation needed early in the year while Basslink was out of service.
  • Basslink has generally flowed north, taking hydro power to the mainland. The interconnector does flow south at times of low hydro output, providing the backup that Tasmania needs (but from brown coal).
  • Hydro generators provide a great synchronous reference, so capacity for extra wind is high.

Victoria

  • Even with imports from Tasmania, Victoria is a net exporter (to SA and NSW). Brown coal is cheap, and will generally displace gas and black coal.
  • Wind and hydro are not insignificant, but are dominated by coal generation.

NEM daily generation

I have generated a new graph of the S.A. blackout and restoration, this time including interconnector flows too (suggestion from James Hazelton).

For the sake of clarity I've clipped negative (SA->V) to 10%. Timeframe is from 00:00 28/9 to 00:00 30/9. Dashed line is at 16:18. All times as AEST Generator Output

The question "Which relay brand is good?" was asked on LinkedIn. A few specific answers were put forth. This is my response, which has been quite well received:

Isn't the best relay the relay that meets your specific needs in the most cost effective and reliable manner? I know this is a non-answer, but every case is going to be different. I've used many of the different brands, and some 'felt' better to me, but other engineers and technicians prefer others.

I reckon the reliability and protection functions are going to be much of a muchness — they'll work, and they'll keep working. What distinguishes the different brands are:

  1. Local support. If you have a problem can you get help in your country in a timely fashion? No point having the best relay if you need to schedule a visit from a support engineer from three countries away.
  2. Free access to programming tools. Some vendors still insist of charging for their programming tools. WHY? Surely the relay is the 'dongle'! My preference is to use the tools that are free to download, and that are easy to use for programming and diagnostics (e.g. event & waveform downloads). Do these tools support data warehouses and revision control systems?
  3. Panel space & wiring. Some relays are HUGE — is this really needed. Can you easily remove a terminal block or slide out the relay if a replacement is needed? Integration flexibility. There is huge variation in how IEC 61850 is implemented by vendors. Do the relays work with logical nodes 'natively' or is IEC 61850 seen as another bolt on protocol like DNP3? Do they support enough report control blocks? Can a dataset be used by more than one RCB or Goose control block?
  4. Documentation — this is linked to (4). Can you get the documents you need to design a system from the vendor and without the need to install all the programming tools? Getting hold of application manuals, MICS, PICS, PIXIT etc all make life easier at the design stage.
  5. Cost. Utilities have budgets to meet, and there is significant variation in pricing (lets not talk details in this forum though). Remember to include support agreements, programming tools, training etc in the costs.

Just my thoughts, and I'd be interested if others expanded on what they used when deciding who to go with for projects or 'panel of supplier' contracts.

The question "Does anyone know if there are guidelines or industry standard for commissioning substation LAN which focuses on Ethernet switches and possibly routers?"" was asked. My response was:

Yang, RFC2544 has a series of tests for benchmarking the performance of interconnect devices like Ethernet switches. The test protocols in there might be a good start for proving network performance.

Any 61850-based substation FAT or SAT should be for the system, not just for the IEDs.

Without Ethernet you don’t have a substation control/protection system. This testing could be system-level performance-based, making sure the inter-trips and controls work within the required time-frame and reliability. I like component testing, but this is more targeted at type testing or design testing. However, if the substation control functional specification has network performance requirements (latency, throughput etc) then this should be tested at the FAT and SAT too.

IEC 61850-4 has the breakdown of what is done and when. FAT and SAT are described in section 7.3.6 and 7.3.7 respectively.


Hi Yang, thanks for the reminder about Y.1564. I had a nagging feeling there was another test spec out there.

I haven’t seen Y.1564 used in substations, but that doesn’t mean it shouldn’t be considered. I think there is much the power industry can learn from Telecoms.

I think a ping test is a very poor test. Firstly it only tests the IP/Management VLAN and not the others that might be in place. Secondly the performance metrics are coarse (millisec times). A more thorough test would include packet injection and capture to make sure frames are not lost at close to line limits. In addition low priority traffic should be injected (unicast and multicast) while protection timing (if GOOSE used for inter-trips) is conducted. Similarly, responsiveness to controls should be tested with some background traffic.

My philosophy is that I want to take the system to breaking point and figure out the safety margin from there. If you don’t break it you don’t know how close to that point you are.

The question was asked "What is the basic difference between PRP (Parallel Redundancy Protocol) and HSR (High availability Seamless Redundancy)". My response was:

Very simply PRP is a duplicate network of whatever design you have (could be rings, mesh or star), while HSR is a ring network. I've seen PRP used in a star configuration, and I think that is its real strength as it is simple.

HSR is the 'zero delay' alternative to RSTP for rings, but does require the use of Redboxes or devices with native support. Quite a few switch manufacturers have embedded modules that provide HSR or PRP functions, or you can purchase code for FPGAs from a number of vendors and use it directly in your device.

@Rod, perhaps in the utility world duplicate protection is the norm, but this is not the case for distribution and industrial applications.

HSR and PRP have application where network based signalling is used and there is a need to maintain the Ethernet switches. With copper signalling each connection is independent, but with Ethernet a switch will affect multiple connections.

RSTP is ambiguous enough with each vendor having differing parameters and their own tweaks for speed. HSR gives more deterministic performance, and PRP avoids slow recovery times with multiple meshes. A second switch is cheaper than duplicating all the protection, and many vendors are supporting HSR/PRP even in their 'low end' devices.

@Rod, I agree that an engineering design (requirements spec, cost/benefit, failure mode study) should be undertaken for any substation LAN design. Unfortunately this doesn't happen.

For small rings RSTP can work fine, and quite a few vendors have dual port IEDs so switches are not required. The hardware cost to go to HSR is minimal (some FPGA code) since there are already two ports on the device. The 'integrated' switch for ring applications has been around for some time (HiPER Ring, MRP, RSTP) — I don't think HSR/PRP has been driving it. For large rings MRP and HSR have definite speed advantages by knowing the topology is a ring. RSTP is more flexible, but that flexibility slows things down.

BTW, as far as integrating protection into a MU — great idea, shame that one of the large European manufacturers is trying to patent this as their own clever idea.

@Juan, differential protection can work through the exchange of phasors too. 64kb/s channels (C37.94 or G.703) are not really enough for raw values, so IEDs process the samples to get the data rate down to a level that can be transmitted.

I think that “Protection Computers” will ultimately do what you propose. ABB already do this with protection of their SVCs, and a Dutch company has a range of products for distribution based on distributed sampling and central protection (but not using 61850). I forget the name at the moment. The substation of the future will have merging units in the field, a few Ethernet switches and two protection computers to do the thinking. The control room can be replaced with a telephone box sized panel!

The question was asked "Does anyone know are there any testing procedures, methodology or testing standards for GOOSE Performance in 61850?". My response was:

The test procedures and methods are given in IEC 61850-10.

The UCAIug also has some test methods, but they are buried in their Sharepoint site. The following tests are listed:

That should give you something to work from, and might avoid the cost of purchasing IEC 61850-10.

The question was asked: "Is is possible to connect different Ethernet switches (Manufacturer / Configuration / Specification) in a ring network ? If yes, then how could be the performance of data transfer?". My response was:

There are a number of international standards that deal with ring topologies. RSTP and MSTP are IEEE standards (802.1D and 802.1Q respectively) that work across vendors. These make a 'virtual break' in the ring that can close when a normally close link opens.

A more reliable (but more complex) alternative is HSR – high-availability seamless redundancy – IEC 62439-3, is vendor independent.

There are vendor specific modifications of standards, such as RuggedCom’s eRSTP that has faster ring repair, or Hirschmann’s HiPER Ring that gives faster ring recovery than RSTP. The Media Redundancy Protocol (MRP), IEC 62439-2, is a derivative of HiPER Ring.

One downside with RSTP is that recovery time increases with the number of devices and especially with the number of meshes. Another is that default parameters may vary between vendors so some tuning is required. RSTP does allow more flexibility in network design than with MRP (which is exclusively for rings).

Performance with HSR and MRP will be deterministic (0 ms for HSR and a defined setting for MRP). RSTP performance will depend on the configured parameters and number of devices, so I would recommend testing of the final design. Software simulation is also a good idea, and something that I don’t see much of in industrial network design.

The real benefit of using a function specific logical node (e.g. PTRC, PDIF etc) is that the LN conveys a lot of semantic information.

If you use GGIO you need to maintain a parallel set of document that describe the purpose of each GGIO. This is really no benefit over DNP3 or Modbus points. If I am subscribing to messages then I know that I have the transduced power (MW) value when I connect CVMMXN1.MX.Watt.mag or PriFouMMXU1.MX.TotW. An analogue GGIO simply doesn’t do this.

Which has more meaning: T3WPDIF1.ST.Op or BOUT_GGIO1.ST.Ind.stVal? I can tell immediately that I've connected the transformer differential trip with the 'proper' LN without needing to refer to anything else. The devices don't care what is subscribed to in the SCD file, but it sure helps reduce the chance of the human making a mistake.

These data attributes come from a real IEC 61860 system. The only reason I used GGIO for outputs is that they came from an RTDS that treats GOOSE output as a binary output card. The IEDs all supported 'real' LNs in their published GOOSE datasets. Making sure I had the right GGIO when setting up IED subscriptions was a pain in the neck—setting up the RTDS subscriptions was very easy by comparison.

IEC 61850 is all about systems, not simply communications between devices.