DRAMSys/README.md

de.uni-kl.ems.dram.vp.system

DRAMSys [1] is a flexible DRAM subsystem design space exploration framework that consists of models reflecting the DRAM functionality, power consumption, temperature behaviour and retention time errors.

Basic Setup

Open a terminal window, go to your home directory, create a directory for your projects and change to it.

$ cd
$ mkdir projects
$ cd projects

Clone the repository.

$ git clone --recursive https://<user>@git.rhrk.uni-kl.de/EIT-Wehn/dram.vp.system.git

The --recursive flag tells git to initialize all submodules within the repository. DRAMPower [2], tinyxml and icewrapper are examples third party repositories that were embedded within the source tree as submodules.

It is possible to work with a fork of the official codebase. In that case, after pushing changes into your fork you should create a pull request in order to get your changes merged into to the official codebase.

$ git clone --recursive https://<user>@git.rhrk.uni-kl.de/<user>/dram.vp.system.git

After cloning go to the project directory.

$ cd dram.vp.system

When working with a fork, the official repository must be added as a remote for your fork.

$ git remote add upstream https://<user>@git.rhrk.uni-kl.de/EIT-Wehn/dram.vp.system.git
$ git remote -v

Also, after a pull request being accepted and merged into the official repository you should get your fork updated.

$ git fetch upstream
$ git checkout master
$ git merge upstream/master
$ git push origin HEAD

Dependencies

Make sure you have properly installed in your system the required libraries. They are:

• SystemC 2.3.1 and TLM 2.0

  The sources can be downloaded from http://accellera.org/downloads/standards/systemc.

  For installation instructions see the installation notes file contained in the release package.

• qwt-6.1.3

  $ svn checkout svn://svn.code.sf.net/p/qwt/code/branches/qwt-6.1
  $ cd qwt-6.1
  $ qmake qwt.pro
  $ make
  $ sudo make install
  

  For further information refer to http://qwt.sourceforge.net/

• python3.4

  In Debian like distros:

  $ sudo apt-get install python3.4
  

• libboost_filesystem and libboost_system

  In Debian like distros:

  $ sudo apt-get install libboost-all-dev
  

• Others

  Some basic libraries may be already installed in your system. If not you can install the following packages or equivalent ones for your distro. In Debian like distros:

  $ sudo apt-get install libc6
  $ sudo apt-get install libstdc++6
  $ sudo apt-get install sqlite3
  $ sudo apt-get install libsqlite3-dev
  $ sudo apt-get install libqt5gui5
  $ sudo apt-get install libqt5sql5
  $ sudo apt-get install libqt5widgets5
  $ sudo apt-get install libqt5core5a
  

To grant flexibility to the user the paths where to find some essential libraries and headers can be specified with environment variables. You can add such variables to you ~/.bashrc file or equivalent.

# SystemC home and target architecture
export SYSTEMC_HOME=<path>
export SYSTEMC_TARGET_ARCH=<[linux,linux64]>

# DRAMSys libraries and headers
export PYTHON_HOME=<path>
export PYTHON_HEADERS=<path>
export LIBQWT_HOME=<path>
export LIBQWT_HEADERS=<path>
export LIBBOOST_HOME=<path>
export LIBBOOST_HEADERS=<path>

Users interested in thermal simulation can also add some extra environment variables:

# Necessary for thermal simulation
export LIBTHREED_ICE_HOME=<path>
export LIBSUPERLU_HOME=<path>

Buiding with QTCreator

Execute the QTCreator.

$ qtcreator &

Use the menu bar and open the DRAMSys project.

File -> Open Project -> dram.vp.system/DRAMSys/dram.vp.system.pro

When you open the project for the first time a configuration window pops-up. Then click in Configure Project and after that Build the project.

To speedup the building process one can use the additional make option -j[jobs]. The command line below returns a good number to be passed to make as the number of jobs that can run simultaneously to improve the building time.

$ cat /proc/cpuinfo | grep processor | wc -l

In the left bar go to Projects -> Build & Run -> Build Steps -> Make. Click in Details then Make arguments and add -j followed by the number you got.

Building without QTCreator

In case you prefer a command line interface to the QTCreator GUI you can also use qmake to generate a Makefile and then compile the project.

$ mkdir build
$ cd build
$ qmake ../DRAMSys/dram.vp.system.pro
$ make

The compilation generates executable binary files dramSys and traceAnalyzer that can be found inside sub-directories.

From the build directory use the commands below to execute DRAMSys.

$ cd simulator
$ ./dramSys

From the build directory use the commands below to execute the traceAnalyzer.

$ cd analyzer
$ ./traceAnalyzer

DRAMSys Thermal Simulation

Before starting make sure you have a clean repository without any previous automatic generated Makefiles. One way to ensure this is by running the command below inside your DRAMSys repository, but keep in mind that untracked files and directories will be removed from the repository.

$ git clean -fdx

This feature can be enabled via an environment variable.

$ export THERMALSIM=true
$ qtcreator &

or

$ mkdir build
$ cd build
$ export THERMALSIM=true
$ qmake ../DRAMSys/dram.vp.system.pro
$ make

The thermal simulation is performed by a 3D-ICE [8] server accessed through the network. Therefore users interested in thermal simulation during their DRAMSys simulations need to make sure they have a 3D-ICE server up and running before starting.

More information about how to obtain 3D-ICE and how to intall it can be found in the official website http://esl.epfl.ch/3D-ICE.

Provided the user fulfilled this installation requisite, before starting DRAMSys it is necessary to run the 3D-ICE server passing to it two arguments: a suitable configuration file and an Internet socket port number. And then wait until the server is ready to receive requests.

$ 3D-ICE-Server <stack file> <port>
Preparing stk data ... done !
Preparing thermal data ... done !
Creating socket ... done !
Waiting for client ... done !

The IP address and the port number related to the server shall be informed in DRAMSys' configuration to subsequent use by DRAMSys to access the thermal simulation server.

Usage Example

The DRAMSys' main configuration file is presented below.

<simulation>
    <!-- General Simulator Configuration (used for all simulation setups) -->
    <simconfig>
        <Debug value="1" />
        <DatabaseRecording value="1" />
        <PowerAnalysis value="1" />
        <NumberOfTracePlayers value="1"/>
        <NumberOfMemChannels value="4"/>
        <ControllerCoreDisableRefresh value="0"/>
        <DynamicTemperatureSimulation value="1"/>
    </simconfig>

    <!-- Temperature Simulator Configuration (used for all simulation setups) -->
    <temperature_simconfig>
        <TemperatureScale value="Celsius" />
        <StaticTemperatureDefaultValue value="23" />
        <DynTemperatureSimPeriod value="100" />
        <DynTemperatureSimUnit value="us" />
        <PowerInfoFile value="../../DRAMSys/simulator/resources/configs/temperature_sim/powerInfo.xml"/>
        <IceServerIp value="127.0.0.1" />
        <IceServerPort value="11880" />
        <SimPeriodAdjustFactor value="10" />
        <NPowStableCyclesToIncreasePeriod value="5" />
        <GenerateTemperatureMap value="1" />
        <GeneratePowerMap value="1" />
    </temperature_simconfig>

    <memspecs>
        <memspec src="../../DRAMSys/simulator/resources/configs/memspecs/WideIO.xml"></memspec>
    </memspecs>

    <addressmappings>
        <addressmapping src="../../DRAMSys/simulator/resources/configs/amconfigs/am_wideio.xml"></addressmapping>
    </addressmappings>

    <memconfigs>
        <memconfig src="../../DRAMSys/simulator/resources/configs/memconfigs/fr_fcfs.xml"/>
    </memconfigs>

    <tracesetups>
        <tracesetup id="fifo">
            <device clkMhz="1000">../../../tests/error/test_error.stl</device>
        </tracesetup>
    </tracesetups>

</simulation>

Enable the error model in fr_fcfs.xml.

<memconfig>
    <BankwiseLogic value="0"/>
    <OpenPagePolicy value="1" />
    <MaxNrOfTransactions value="8" />
    <Scheduler value="FR_FCFS" />
    <Capsize value="5" />
    <PowerDownMode value="NoPowerDown"/>
    <PowerDownTimeout value="100" />
    <!-- Error Model: -->
    <ErrorChipSeed value="42" />
    <ErrorCSVFile value="../../DRAMSys/simulator/src/error/error.csv" />
    <ErrorStoreMode value="ErrorModel" /> <!--3 Modes: NoStorage, Store (store data without errormodel), ErrorModel (store data with errormodel)-->
</memconfig>

Generate the input trace file for DRAMSys.

$ cd DRAMSys/tests/error/
$ ./generateErrorTest.pl > test_error.stl 

Start the 3D-ICE server providing the stack file and the port number.

$ cd DRAMSys/simulator/resources/configs/temperature_sim
$ 3D-ICE-Server stack.stk 11880

In another terminal or terminal tab start DRAMSys. Here the program's output is redirected to a file.

$ cd build/simulator/
$ ./dramSys > output

DRAMSys Configuration

The dramSys executable supports one argument which is a XML file that contains configurable aspects of the desired simulation. If no argument is passed through the command line a default configuration file will be loaded.

The XML code below shows a typic configuration:

<simulation>

    <!-- General Simulator Configuration (used for all simulation setups) -->
    <simconfig>
        <Debug value="1"/>
        <DatabaseRecording value="1"/>
        <PowerAnalysis value="1"/>
        <NumberOfTracePlayers value="5"/>
        <NumberOfMemChannels value="1"/>
        <ControllerCoreDisableRefresh value="0"/>
        <DynamicTemperatureSimulation value="0"/>
    </simconfig>

    <!-- Temperature Simulator Configuration (used for all simulation setups) -->
    <temperature_simconfig>
        <TemperatureScale value="Celsius" />
        <StaticTemperatureDefaultValue value="89" />
        <DynTemperatureSimPeriod value="100" />
        <DynTemperatureSimUnit value="ms" />
        <PowerInfoFile value="../../DRAMSys/simulator/resources/configs/temperature_sim/power_thresholds.xml"/>
        <IceServerIp value="127.0.0.1" />
        <IceServerPort value="11880" />
        <SimPeriodAdjustFactor value="10" />
        <NPowStableCyclesToIncreasePeriod value="5" />
        <GenerateTemperatureMap value="1" />
        <GeneratePowerMap value="1" />
    </temperature_simconfig>

    <!-- Memory Specifications -->
    <memspecs>
        <memspec src="../../DRAMSys/simulator/resources/configs/memspecs/WideIO.xml"></memspec>
    </memspecs>

    <!-- Address Mappings -->
    <addressmappings>
        <addressmapping src="../../DRAMSys/simulator/resources/configs/amconfigs/am_wideio.xml"></addressmapping>
    </addressmappings>

    <!-- Memory Configurations -->
    <memconfigs>
        <memconfig src="../../DRAMSys/simulator/resources/configs/memconfigs/fifo.xml"/>
    </memconfigs>
    
    <!-- Trace Setups -->
    <tracesetups>
        <!-- Multiple trace setups are allowed for the same simulation setup -->
        <tracesetup id="fifo">
            <!--
            Specify here a trace file for each of the trace players. Trace
            players without a file will not generate transactions.
            It is also possible to choose "cklMhz" and the "bl" for every
            player.
            -->
            <device clkMhz="200">voco2.stl</device>
            <device clkMhz="200">voco2.stl</device>
            <device clkMhz="200">voco2.stl</device>
        </tracesetup>
    </tracesetups>

</simulation>

Some configuration fields reference other XML files which contain more specialized chunks of the configuration like memory specification, address mapping and memory configurations.

The XML configuration files are parsed by the program and the configuration details extracted are assigned to the correspondent attributes of the internal configuration structure.

Simulation Setups

Every possible combination of memory specification, address mapping and memory configuration corresponds to a simulation setup.

DRAMSys executes all the trace setups listed in the configuration file for each of the simulation setups.

A single trace setup is composed of an id string and one or more devices.

The device configuration consists of two configuration fields - clkMhz (operation frequency for this device) and bl (burst length) - and a trace file.

A trace file is a pre-recorded file containing memory transactions. Each memory transaction has a timestamp that tells the simulator when it shall happen, a transaction type (e.g. read, write) and a memory address.

A trace player is equivalent to a bus master device (i.e. a device that locks a bus and generates memory transactions). By adding device elements into the trace setup section one can specify the operation frequency, the burst length and the trace file to be used by trace players.

Trace players without a corresponding device configuration do not generate transactions.

DRAMSys executes all the simulation setups within the configuration file providing flexibility for exhaustive explorations.

Configuration File Sections

The main configuration file is divided into self-contained sections, each of these sections is a set of logically related configuration aspects for the simulation.

Below are listed the configuration sections and configuration fields.

• Simulator Configuration

  • Debug (boolean)
    • "1": enables debug output on console
    • "0": disables debug output
  • DatabaseRecording (boolean)
    • "1": enables trace file recording for the trace analyser tool
    • "0": disables trace file recording
  • PowerAnalysis (boolean)
    • "1": enables live power analysis with the DRAMPower tool
    • "0": disables power analysis
  • NumberOfTracePlayers (unsigned int)
    • Number of trace players
  • NumberOfMemChannels (unsigned int)
    • Number of memory channels
  • ControllerCoreDisableRefresh (boolean)
    • "1": disables refreshes
    • "0": normal operation (refreshes enabled)
  • DynamicTemperatureSimulation (boolean)
    • "1": enables the dynamic temperature simulation feature
    • "0": static temperature during simulation

• Temperature Simulator Configuration

  • TemperatureScale (string)
    • "Celsius"
    • "Fahrenheit"
    • "Kelvin"
  • StaticTemperatureDefaultValue (int)
    • Temperature value for simulations with static temperature
  • DynTemperatureSimPeriod (double)
    • Period of the dynamic temperature simulation
  • DynTemperatureSimUnit (string)
    • "s": seconds
    • "ms": millisecond
    • "us": microseconds
    • "ns": nanoseconds
    • "ps": picoseconds
    • "fs": femtoseconds
  • PowerInfoFile (string)
    • File containing power related information: devices identifiers, initial power values and power thresholds.
  • IceServerIp (string)
    • 3D-Ice server IP address
  • IceServerPort (unsigned int)
    • 3D-Ice server port
  • SimPeriodAdjustFactor (unsigned int)
    • When substantial changes in power occur (i.e., changes that exceed the thresholds), then the simulation period will be divided by this number causing the thermal simulation to be executed more often.
  • NPowStableCyclesToIncreasePeriod (unsigned int)
    • Wait this number of thermal simulation cycles with power stability (i.e., changes that do not exceed the thresholds) to start increasing the simulation period back to its configured value.
  • GenerateTemperatureMap (boolean)
    • "1": generate temperature map files during dynamic temperature simulation
    • "0": do not generate temperature map files during dynamic temperature simulation
  • GeneratePowerMap (boolean)
    • "1": generate power map files during dynamic temperature simulation
    • "0": do not generate power map files during dynamic temperature simulation

• Memory Specification

  A file with memory specifications. This information comes from datasheet and usually does not change.

• Address Mapping

  XML files describe the address mapping to be used in the simulation.

  The file am_wideio.xml is a good example.

  <!-- Row Bank Column -->
  
  <addressmapping>
      <channel from="27" to="28" />
      <row   from="14" to="26" />		
      <bank  from="11" to="13" />
      <column from="4" to="10" />
      <bytes from="0" to="3" /> 
  </addressmapping>
  

  

  <!-- Bank Row Column -->
  
  <addressmapping>
      <channel from="27" to="28" />
      <bank  from="24" to="26" />
      <row   from="11" to="23" />
      <column from="4" to="10" />
      <bytes from="0" to="3" />
  </addressmapping>
  

  

• Memory Configuration

  The content of fifo.xml is presented below as an example.

  <memconfig>
    <BankwiseLogic value="0"/>
    <OpenPagePolicy value="1"/>
    <MaxNrOfTransactions value="8"/>
    <Scheduler value="FIFO_STRICT"/>
    <Capsize value="5"/>
    <PowerDownMode value="NoPowerDown"/>
    <PowerDownTimeout value="100"/>
    <!-- Error Modelling -->
    <ErrorChipSeed value="42"/>
    <ErrorCSVFile value="../../DRAMSys/simulator/src/error/error.csv" />
    <ErrorStoreMode value="NoStorage"/>
  </memconfig>
  

  • BankwiseLogic (boolean)
    • "1": perform bankwise-refresh [3] and bankwise-powerdown [4]
    • "0": do not perform bankwise operations
  • OpenPagePolicy (boolean)
    • "1": use open page precharge policy
    • "0": do not use open page precharge policy
  • MaxNrOfTransactions (unsigned int)
    • Maximum number of transactions.
  • Scheduler (string)
    • "FIFO": first in, first out
    • "FIFO_STRICT": out-of-order treatment of queue elements not allowed
    • "FR_FCFS": first-come, first-served
  • Capsize (unsigned int)
    • Capacitor cell size.
  • PowerDownMode (enum EPowerDownMode)
    • "NoPowerDown": no power down mode (active idle)
    • "Staggered": staggered power down policy [5]
    • "TimeoutPDN": precharge idle
    • "TimeoutSREF": self refresh
  • Buswidth (unsigned int)
    • Bus width in bits.
  • ReadWriteGrouping (boolean)
    • "1": enable read writing grouping
    • "0": disable read writing grouping
  • ReorderBuffer (boolean)
    • "1": use reordering buffer
    • "0": do not use reordering buffer
  • ErrorChipSeed (unsigned int)
    • Seed to initialize the random error generator.
  • ErrorCSVFile (string)
    • CSV file with error injection information.
  • ErrorStoreMode (enum ErrorStorageMode)
    • "NoStorage": no storage
    • "Store": store data without error model
    • "ErrorModel": store data with error model [6]

• Trace Setups

  • id (string)
    • Trace setup id. Two kinds of output files are generated by DRAMSys: SQLite databases containing transactions related to each memory channel (.tdb) and a text file (.txt) with the program output. The base name for these files comes from this field.
  • clkMhz (unsigned int)
    • Speed of the trace player
  • bl (unsigned int)
    • Burst length
  • trace file
    • A pre-recorded file containing memory transactions to be executed by a trace player.

Some attributes are self-explanatory while others require some previous knowhow of memory technologies.

Resources of the simulator are available inside of the resources directory and its sub-directories.

$ cd DRAMSys/simulator/resources

A description of the content each directory follows.

• resources
  • configs: XML files used for configure specific details of the simulation.
    • am_configs: address mapping configuration
    • memconfigs: memory configuration
    • memspecs: configuration related to the memory technology
  • scripts: useful tools like address scrambler, trace analyser, database creator, etc.
  • simulations: global configuration
  • traces: trace files for simulations. They contain accesses to memory in certain known scenarios.

References

[1] TLM Modelling of 3D Stacked Wide I/O DRAM Subsystems, A Virtual Platform for Memory Controller Design Space Exploration M. Jung, C. Weis, N. Wehn, K. Chandrasekar. International Conference on High-Performance and Embedded Architectures and Compilers 2013 (HiPEAC), Workshop on: Rapid Simulation and Performance Evaluation: Methods and Tools (RAPIDO), January, 2013, Berlin.

[2] DRAMPower: Open-source DRAM Power & Energy Estimation Tool Karthik Chandrasekar, Christian Weis, Yonghui Li, Sven Goossens, Matthias Jung, Omar Naji, Benny Akesson, Norbert Wehn, and Kees Goossens URL: http://www.drampower.info

[3] Energy Optimization in 3D MPSoCs with Wide-I/O DRAM M. Sadri, M. Jung, C. Weis, N. Wehn, L. Benini. Conference Design, Automation and Test in Europe (DATE), March, 2014, Dresden, Germany.

[4] DRAMSys: A flexible DRAM Subsystem Design Space Exploration Framework M. Jung, C. Weis, N. Wehn. Accepted for publication, IPSJ Transactions on System LSI Design Methodology (T-SLDM), October, 2015.

[5] Optimized Active and Power-Down Mode Refresh Control in 3D-DRAMs M. Jung, M. Sadri, C. Weis, N. Wehn, L. Benini., VLSI-SoC, October, 2014, Playa del Carmen, Mexico.

[6] Retention Time Measurements and Modelling of Bit Error Rates of WIDE-I/O DRAM in MPSoCs C. Weis, M. Jung, P. Ehses, C. Santos, P. Vivet, S. Goossens, M. Koedam, N. Wehn. Accepted for publication, IEEE Conference Design, Automation and Test in Europe (DATE), March, 2015, Grenoble, France

[7] http://www.uni-kl.de/3d-dram/publications/

[8] A Sridhar, A Vincenzi, D Atienza, T Brunschwiler, 3D-ICE: a compact thermal model for early-stage design of liquid-cooled ICs, IEEE Transactions on Computers (TC 2013, accepted for publication).

[9] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, 3D-ICE: Fast compact transient thermal modeling for 3D-ICs with inter-tier liquid cooling, Proceedings of the 2010 International Conference on Computer-Aided Design (ICCAD 2010), San Jose, CA, USA, November 7-11 2010.

[10] A Sridhar, A Vincenzi, M Ruggiero, T Brunschwiler, D Atienza, Compact transient thermal model for 3D ICs with liquid cooling via enhanced heat transfer cavity geometries, Proceedings of the 16th International Workshop on Thermal Investigations of ICs and Systems (THERMINIC'10), Barcelona, Spain, 6-8 October, 2010.

[11] http://esl.epfl.ch/3D-ICE