This questionnaire is prepared to gather information on verification and validation (V&V) activities for computational simullations in scientific work in areas of computational mechanics.
It consists of 3 parts:
1) About you and your projects;
2) V&V activities;
3) Data used for V&V.
It will take You from 15 to 30 min. to fill it.
The questionnaire is annonymous. The summary results will be published on http://mlewicka-mgr.blogspot.com until the end of May.
Should You have any questions please contact magdalena.lewicka [at] gmail.com
Project Overview
Here are some general questions about you and your projects. Please, relate them to your typical project or to the last one.
General information
What is your position in a project (general, connected with the university position)?
* Research/Project Assistant Project Leader/Supervisor (mostly managing, controlling than executing) Other Large scale simulation can be considered as computational simulation consisted on tens, hundreds, thousands or more equations to solve.
Estimate your experience in large scale simulation projects (in months, 1 month:= 6h*4(days)*4(weeks) = 96h)
* < 6 6 - 24 > 24 Do you programm?
* Yes No For how long have you programmed? (in months, 1 month:=96h)
* < 6 6 - 24 > 24 Which computer languages do you use?
* Fortran C/C++ Java,C# Script languages (e.g. Python, TCL, Ruby) Mathematical programming languages (e.g. Mathlab, Octave, Mathematica, Maple) Other How many hours of courses (programming courses, software engineering related topics) have you taken up?
* 0 15 - 30 31 - 90 > 90
Projects overview
What are the reasons for your project?
* Part of my work For science needs For commercial needs Other What are general goals of your project?
* Invention Improvement Evaluation Other Choose general types of projecs that you have been involved in:
* Modelling/ Designing projects - project outcome is a design of a construction/ real-world element Software projects - project outcome is an application Projects based on experiments - project deliverable is an experiment environment and its results Analytic projects - project deliverable is an analytical solution (including methods and results)/ mathematical model None of them, other type How many people at the university are involved in your project?
* 1 person - just me 2 people: me and sb else (supervisor or project assistant) 3 people and more Estimate your time (in %) in last project spent on:
Do you use any version controll system (e.g. CVS, SVN) during your work?
* Yes No
Verification and Validation Processes
This section refers to verification and validation (V&V) aspects of your projects. Please, relate all questions to your typical project or to your last one. VERIFICATION can be treated as a comparison between two issues (e.g. 2 solutions, 2 models) where none of issues is a part of real world. Whereas VALIDATION can be concerned as a comparison between two issues, where one of issues is a part of real world (e.g. experiment results, real model behaviour).
Verification and Validation Plan
Below are few questions about defining verification and validation (V&V) processes for your poject.
Does your project plan include V&V plan?
* Yes, and it is a detailed specification Yes, but it is a general plan Sometimes Very rarely No, because I don't verify or validate my models/software/results No, because I usually verify or validate with the same methods No, for other reasons What issues does your V&V plan include?
* List of key points to verify List of key points to validate List of experiments to carry out Description of errors to estimate Description of V&V methods Benchmarks and samples to test code Software or hardware configuration (e.g. properties of hardware, list of tools/ technologies) to help V&V processes Other Why don't you have a V&V specification for your project?
Verification and Validation Activities
Solution verification
Do you estimate the numerical accuracy of the given solution?
* Yes No How do you estimate the accuracy of the solution?
* Comparison of numerical solutions of the discretized equations on different grid sizes (related to h-adaptivity) Comparison of numerical solutions from different discretization methods on the same grid (related to p-adaptivity) Other Why don't you estimate the accuracy of the solution? Which of the following errors do you estimate?:
* Discretization error (related to the numerical method used to solve the problem) Iterative error (related to approximation error) Geometry approximations errors Other None
Code Verification
This part refers to the software, programs which you have written during the project.
Numerical Model/Algorithm Verification
How do you verify implemented numerical model/ algorithm?
* Analytic solutions for simplified physics Method of manufactured solution Benchmark solutions Other Describe your other method:
Software Quality Analysing and Testing
Which of the following static analysis activities do you do?
* Syntax Analysis (loops; conditions; variables types; function inputs-outputs types) Data Analysis (data dependiency; data flow by making a flow diagram and analysing it) Cause-Effect graphing Controll analysis (calling structure; concurrent process; control flow; state transition) Interface analysis (model interface; user inteface) Fault/Failure analysis None of them Which of the following dynamic analysis activities do you do? Unit tests (e.g. assertion checking) Black Box tests (functional tests): output examining under a variety of input parameters White Box tests (structural tests): analysing of conditions, data flow, loops, paths Special Input testing (examining the boundary values, extreme/ invalid/ real-time/ self-driven input) Debugging Printing the input parameters at the end of the simulation, to be sure that they are not changed Bottom-Up testing Fault/Failure Insertion testing Visualization/Animation of running program Graphical Comparisons Sensitivity analysis Interface testing (data; model; user) Object-Flow testing Printing out a wide variety of output statistics Complaince testing (authorisation; security; standards; performance) None of them
Software Quality Assurance Practises
Here are few questions which refers to practises that help to keep your code easy to verify.
Is your code as self-documenting as possible? (precise definition of every variable used and a general description of the purpose of each major section of code)?
* Yes No Is your code checked by someone else than you?
* Yes No Do you use Design Patterns?
* Yes No Iterative and incremental methodology is a software development process. Whole process is divided in several steps, which are standalone and delivere a part of functionality of your software. Examples of methodologies: XP, SCRUM, Agile and many more adopted to project needs.
Do you use iterative methodologies for programming?
* Yes No Do you write a test first before writing the code?
* Yes No Do you learn other programming technologies or languages at least once a year?
* Yes No
Conceptual Model Validation
Which of the following activities do you do?
* Building the conceptual model basing on empirical data Theoretical derivation Personal Communication (asking people whether the model and/or its behaviour is reasonable; it requires flow chart, graphical model) Applying accepted approaches, making rational decisions while building and judging the model Validation with some historical data Applying "positive economics" (:= an approach that requires only the model to be able to predict the future effects and is not concerned with model assumptions and structure) Traces (following specific entities of model to determine if the logic is correct and if the necessary accuracy is obtained) Other None of them
Numerical Model Validation
What kind of experiments do you carry out?
* Experiments that are conducted primarily to improve the fundamental understanding of some phisical process. Sometimes these are referred to as physical-discovery experiments Experiments that are conducted primarilyfor construction or improving mathematical models of fairly well-understood phisical processes Experiments that determine or improve reliability, performance, or safety of components, subsystems, or complete systems Experiments that are designed, executed, and analysed for the purpose of quantitatively determining the ability of a mathematical model and its embodiment (=representative model) in a computer code to simulate a well- characterized physical process I don't do experiments
What is the most difficult in your project to verify or validate?
* To validate models which integrate many different components, so a qualitative dependency on input parameters is not obvious To verify modelling results which are the solution of coupled processes which you can't decouple. There is no sure, where the deviation come from. To verify whether the assumptions and simplifications which were done while modelling, are justified or not To verify accuracy of the computer model To validate The numerical formulation underlying physical model Other I have no problems with V&V Describe other difficulity: Do you document your V&V results (including testing results if you programm) ?
* Yes No Do you write a V&V summary report for your supervisor/customer/project leader (or do you get such a report from your team if you are a supervisor)?
* Yes No
Data for Verification and Validation
This section is about acquiring, maintaining and transforming data needed to execute necessary V&V activities in your project.
Data for V&V
How do you acquire data for V&V use?
* From the Internet From universities' repositories From libraries From specialistic benchmark databases From a customer/company I prepare data by myself using free/commercial solvers Other If possible, please, give the address of the benchmark database: Finding data for comparison tests for your engineering's problems is in your opinion:
* Very difficult Rather difficult Rather easy Very easy Have you ever had to rewrite numerical data (e.g. from tables) from articles and other sources to prepare data for V&V?
* Yes No Have you ever had to reproduct graphs, figures, grids etc. from publications and other sources in order to compare them with your results and publish those comparisons? Yes No If the data (e.g. graphs, tables) was available electronically (on-line) in editable formats how much it would help you with work?
* Unearthly Very Without special influence How often, in your opinion, scientific publications don't contain complete information for equivalent reproduction of the presented results?
* Very often Often Sometimes Hardly ever Never Have you ever had a situation, that your simulation results were completely different from those presented in the literature?
* Yes No Have you ever had to write a program/script for format conversion: from data format published in a paper to a format which is used by your tools/software?
* Yes No How difficult is writing programs for format conversion? Very difficult Rather difficult Rather easy Very easy Do you prepare verification/validation benchmarks by yourself?
* Yes No Where do you keep your benchmarks?
* On my computer only I public them on my/university website only I put them in some on-line official repositories/databases What should a benchmark solution contain to be the most useful to you?
* Conceptual description Mathematical description Accuracy assessment Experimental description Description of uncertain quantification of benchmark measurements Code/scripts used for solved problems Figures, tables etc. in editable formats Other Describe other item that a benchmark should contain: Please, once again estimate your time (in %) in last finished project spent on V&V activities (please don't try to recreate your previous answer) :
*
Free text area for your opinion, suggestions and other statements, if you wish.
